2024-03-11T18:08:32.757 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T18:08:32.758 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226 --verbose --name teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/orig.config.yaml 2024-03-11T18:08:32.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T18:08:32.919 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591226' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 94a539707733325322964c6a187c6a1b41be1daf subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad owner: scheduled_teuthology@teuthology priority: 250 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8179 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 297/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b6a4997be2b8bb74411268cc42753f29ae9a78ad syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvGf+cyMZ6EdEn2J5RHTQc6jlKSmHOa15J+g16R020D/WnTP8sd+p/577nli/j7ejRB8I2iXIfrbjR9bK02/Ro= smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtVfIZXn/vKDzdE72Fnb01pPZmHprqSHaw9vDG/7pOEIskG0900+I48l30rMFl6AWLRMiO2ef3/L3kCdI0AZ3U= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDICgzGvuK60VEuW89K0LVqhLtrL7SvthPpgehstdTNAi0NW8PNpL1kCZN4Lea83uh8HZQWUTcO9hc//CYBm1sc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_14:02:35 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T18:08:32.920 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa; will attempt to use it 2024-03-11T18:08:32.921 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks 2024-03-11T18:08:32.921 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T18:08:32.923 INFO:teuthology.task.internal:Checking packages... 2024-03-11T18:08:32.946 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T18:08:32.947 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T18:08:32.947 INFO:teuthology.packaging:ref: None 2024-03-11T18:08:32.947 INFO:teuthology.packaging:tag: None 2024-03-11T18:08:32.947 INFO:teuthology.packaging:branch: squid 2024-03-11T18:08:32.947 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:08:32.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T18:08:33.096 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T18:08:33.098 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T18:08:33.104 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T18:08:33.104 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T18:08:33.113 INFO:teuthology.task.internal:Saving configuration 2024-03-11T18:08:33.134 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T18:08:33.141 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T18:08:33.169 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.913386', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvGf+cyMZ6EdEn2J5RHTQc6jlKSmHOa15J+g16R020D/WnTP8sd+p/577nli/j7ejRB8I2iXIfrbjR9bK02/Ro='} 2024-03-11T18:08:33.196 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.917054', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtVfIZXn/vKDzdE72Fnb01pPZmHprqSHaw9vDG/7pOEIskG0900+I48l30rMFl6AWLRMiO2ef3/L3kCdI0AZ3U='} 2024-03-11T18:08:33.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.915329', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDICgzGvuK60VEuW89K0LVqhLtrL7SvthPpgehstdTNAi0NW8PNpL1kCZN4Lea83uh8HZQWUTcO9hc//CYBm1sc='} 2024-03-11T18:08:33.216 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T18:08:33.223 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-11T18:08:33.223 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-11T18:08:33.223 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-11T18:08:33.224 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T18:08:33.323 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f483ded8160>, signals=[15]) 2024-03-11T18:08:33.323 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T18:08:33.329 INFO:teuthology.task.internal:Opening connections... 2024-03-11T18:08:33.329 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:08:33.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:08:33.438 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T18:08:33.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:08:33.545 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-03-11T18:08:33.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:08:33.634 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T18:08:33.641 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-03-11T18:08:33.659 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-03-11T18:08:33.660 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-03-11T18:08:33.718 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-03-11T18:08:33.718 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:08:33.719 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:08:33.720 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-03-11T18:08:33.760 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-11T18:08:33.779 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-11T18:08:33.779 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-11T18:08:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-11T18:08:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-11T18:08:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-11T18:08:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:08:33.836 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:08:33.837 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:08:33.837 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-11T18:08:33.868 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-03-11T18:08:33.886 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-03-11T18:08:33.886 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-03-11T18:08:33.943 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T18:08:33.944 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-03-11T18:08:33.944 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T18:08:33.944 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T18:08:33.944 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T18:08:33.944 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-03-11T18:08:33.968 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T18:08:33.979 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T18:08:34.016 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T18:08:34.016 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:08:34.018 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:08:34.021 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-03-11T18:08:34.036 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T18:08:34.045 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T18:08:34.045 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:08:34.076 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:08:34.078 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-03-11T18:08:34.106 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T18:08:34.191 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T18:08:34.208 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-11T18:08:34.208 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-11T18:08:34.208 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-11T18:08:34.208 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:08:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:08:34.226 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-11T18:08:34.439 DEBUG:teuthology.task.kernel:sha1 for client.0 is dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T18:08:34.439 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412"... 2024-03-11T18:08:34.439 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-03-11T18:08:34.457 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-361.el9.x86_64 2024-03-11T18:08:34.458 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T18:08:34.458 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-11T18:08:34.459 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2024-03-11T18:08:36.609 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T18:08:38.100 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-11T18:08:39.099 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-11T18:08:41.175 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-11T18:08:45.534 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-11T18:08:47.127 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-11T18:08:48.608 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-11T18:08:53.761 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-11T18:08:56.150 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:08:56.165 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-03-11T18:08:56.166 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-03-11T18:08:56.166 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 1 Package 2024-03-11T18:08:56.166 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:08:56.166 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 375 M 2024-03-11T18:08:56.166 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-03-11T18:09:01.187 INFO:teuthology.orchestra.run.smithi002.stdout:linux-firmware-20240219-143.el9.noarch.rpm 75 MB/s | 375 MB 00:05 2024-03-11T18:09:01.188 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:09:01.189 INFO:teuthology.orchestra.run.smithi002.stdout:Total 75 MB/s | 375 MB 00:05 2024-03-11T18:09:01.189 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-03-11T18:09:01.279 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-03-11T18:09:01.279 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-03-11T18:09:03.776 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-03-11T18:09:03.776 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-03-11T18:09:10.369 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-03-11T18:09:10.607 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-11T18:09:10.697 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T18:09:12.110 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T18:09:12.111 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-11T18:09:12.519 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-11T18:09:12.520 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:09:12.521 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-03-11T18:09:12.521 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-11T18:09:12.521 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:09:12.521 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-03-11T18:09:12.820 INFO:teuthology.task.kernel:Downloading kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-11T18:09:12.821 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-11T18:09:12.966 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ 2024-03-11T18:09:12.966 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-11T18:09:12.969 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2024-03-11T18:09:24.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11 18:09:24 URL:https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm [336216379/336216379] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-11T18:09:24.393 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2024-03-11 18:09:24-- 2024-03-11T18:09:24.393 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 11s 2024-03-11T18:09:24.393 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 321M in 11s (28.8 MB/s) 2024-03-11T18:09:24.397 INFO:teuthology.task.kernel:Installing kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-11T18:09:24.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-11T18:09:26.432 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2024-03-11T18:09:26.489 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2024-03-11T18:09:26.573 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2024-03-11T18:09:34.444 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-6.8.0_rc7_gdd5c1cd17167-1 ######################################## 2024-03-11T18:09:50.078 INFO:teuthology.orchestra.run.smithi002.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T18:10:13.205 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-11T18:10:13.206 DEBUG:teuthology.orchestra.run.smithi002:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-11T18:10:13.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/System.map 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/config 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-11T18:10:13.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-11T18:10:13.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ansi_cprng.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/anubis.ko 2024-03-11T18:10:13.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/arc4.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_memcpy.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_pq.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_tx.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_xor.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/raid6test.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authenc.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authencesn.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blake2b_generic.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_common.ko 2024-03-11T18:10:13.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/camellia_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast5_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast6_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast_common.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ccm.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cmac.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crc32_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_engine.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_user.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cts.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/des_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/dh_generic.ko 2024-03-11T18:10:13.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/echainiv.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/essiv.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/fcrypt.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/khazad.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4hc.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/md4.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/michael_mic.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcbc.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcrypt.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/rmd160.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/seed.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/serpent_generic.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tcrypt.ko 2024-03-11T18:10:13.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tea.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_common.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_generic.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/vmac.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/wp512.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xcbc.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xor.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xxhash_generic.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_pad.ko 2024-03-11T18:10:13.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/custom_method.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/ec_sys.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/platform_profile.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbs.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbshc.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/video.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/acard-ahci.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ahci_platform.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ata_generic.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/libahci_platform.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_acpi.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ali.ko 2024-03-11T18:10:13.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_amd.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_artop.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atiixp.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atp867x.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd640.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd64x.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cypress.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_efar.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt366.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt37x.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-11T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it8213.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it821x.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_jmicron.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_marvell.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_mpiix.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_netcell.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ninja32.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87410.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87415.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_oldpiix.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_opti.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_optidma.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pcmcia.ko 2024-03-11T18:10:13.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_piccolo.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_rdc.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sch.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_serverworks.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sil680.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sis.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sl82c105.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_triflex.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_via.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pdc_adma.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_inic162x.ko 2024-03-11T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_mv.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_nv.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_promise.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_qstor.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil24.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sis.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_svw.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sx4.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_uli.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_via.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_vsc.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay/ks0108.ko 2024-03-11T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma/bcma.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe/aoe.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/brd.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd/drbd.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/floppy.ko 2024-03-11T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/loop.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/nbd.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk/null_blk.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/pktcdvd.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/rbd.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/virtio_blk.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkfront.ko 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram 2024-03-11T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram/zram.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hangcheck-timer.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/via-rng.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-11T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/lp.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave/mwave.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ppdev.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tlclk.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-11T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/uv_mmtimer.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/virtio_console.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp 2024-03-11T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-aes.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-sha.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca/dca.ko 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq 2024-03-11T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/amd64_edac.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/e752x_edac.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/edac_mce_amd.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3000_edac.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3200_edac.ko 2024-03-11T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5100_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5400_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7300_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7core_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i82975x_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/ie31200_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/sb_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/x38_edac.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/edd.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio 2024-03-11T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-ich.ko 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-appleir.ko 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-aureal.ko 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-axff.ko 2024-03-11T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-dr.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elecom.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elo.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-emsff.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gaff.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gt683r.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gyration.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtekff.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-hyperv.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-icade.ko 2024-03-11T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-keytouch.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-kye.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lcpower.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-led.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lenovo.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-multitouch.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-ortek.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-penmount.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-petalynx.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-picolcd.ko 2024-03-11T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-pl.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-plantronics.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-primax.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-rmi.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-common.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-11T18:10:13.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-saitek.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-samsung.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sjoy.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sony.ko 2024-03-11T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-speedlink.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-steelseries.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sunplus.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tivo.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tmff.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-topseed.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-twinhan.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-uclogic.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-waltop.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-wiimote.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-xinmo.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zpff.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zydacron.ko 2024-03-11T18:10:13.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/uhid.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/wacom.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_balloon.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_utils.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_vmbus.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru3.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7414.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7418.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adc128d818.ko 2024-03-11T18:10:13.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1025.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1026.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1029.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1031.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm9240.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ads7828.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7410.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7411.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7462.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7470.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7475.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7x10.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/amc6821.ko 2024-03-11T18:10:13.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/applesmc.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asb100.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asc7621.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/atxp1.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/coretemp.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dme1737.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds1621.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds620.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc1403.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc6w201.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71805f.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71882fg.ko 2024-03-11T18:10:13.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f75375s.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fam15h_power.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fschmd.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g760a.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g762.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl518sm.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl520sm.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5500_temp.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5k_amb.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmaem.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmpex.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina209.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina2xx.ko 2024-03-11T18:10:13.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/it87.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k10temp.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k8temp.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lineage-pem.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm63.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm73.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm75.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm77.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm78.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm80.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm83.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm85.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm87.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm90.ko 2024-03-11T18:10:13.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm92.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm93.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95234.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95241.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95245.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc2945.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4151.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4215.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4222.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4245.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4260.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4261.ko 2024-03-11T18:10:13.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max16065.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1619.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1668.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max197.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6639.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6650.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6697.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/mcp3021.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6683.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775-core.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct7802.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87360.ko 2024-03-11T18:10:13.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87427.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pcf8591.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-11T18:10:13.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/powr1220.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5627.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5636.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht15.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht21.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/shtc1.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sis5595.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47b397.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m1.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m192.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/thmc50.ko 2024-03-11T18:10:13.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp102.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp103.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp401.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp421.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via-cputemp.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via686a.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt1211.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt8231.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627ehf.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627hf.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83781d.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83791d.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83792d.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83793.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83795.ko 2024-03-11T18:10:13.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l785ts.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l786ng.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-11T18:10:13.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-11T18:10:13.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-dev.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-mux.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-smbus.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-stub.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_core.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-11T18:10:13.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp 2024-03-11T18:10:13.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-11T18:10:13.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/gameport.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/joydev.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/appletouch.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/bcm5974.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/cyapatp.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/sermouse.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-11T18:10:13.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/altera_ps2.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/arc_ps2.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/serio_raw.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/sparse-keymap.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-blinkm.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lm3530.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lp3944.ko 2024-03-11T18:10:13.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-ss4200.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache/bcache.ko 2024-03-11T18:10:13.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-bio-prison.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache-smq.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-crypt.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-delay.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-flakey.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-log-userspace.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-multipath.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-queue-length.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-raid.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-round-robin.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-service-time.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-switch.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-thin-pool.ko 2024-03-11T18:10:13.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-verity.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid0.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid1.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid10.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid456.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptbase.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptctl.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptfc.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptlan.ko 2024-03-11T18:10:13.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptsas.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptscsih.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptspi.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_ich.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_sch.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/mfd-core.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/sm501.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/viperboard.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/vx855.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/wl1273-core.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-11T18:10:13.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds9802als.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds990x.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/bh1770glc.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710/cb710.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/at24.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/max6875.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/enclosure.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/hpilo.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29003.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29020.ko 2024-03-11T18:10:13.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-me.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-txe.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-11T18:10:13.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_7xx1.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_core.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tsl2550.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_balloon.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding/bonding.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-11T18:10:13.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dummy.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/eql.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-11T18:10:13.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-11T18:10:13.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-11T18:10:13.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-11T18:10:13.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-11T18:10:13.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dnet.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ethoc.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/fealnx.ko 2024-03-11T18:10:13.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf 2024-03-11T18:10:13.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/jme.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-11T18:10:13.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia 2024-03-11T18:10:13.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-11T18:10:13.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-11T18:10:13.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-11T18:10:13.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/geneve.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154 2024-03-11T18:10:13.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ifb.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvlan.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvtap.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mii.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/net_failover.ko 2024-03-11T18:10:13.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/netconsole.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/nlmon.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/amd.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm87xx.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/broadcom.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/cicada.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/davicom.ko 2024-03-11T18:10:13.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/dp83640.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/et1011c.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/icplus.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/lxt.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/marvell.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/micrel.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/national.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/phylink.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/qsemi.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/realtek.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/smsc.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/ste10Xp.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/vitesse.ko 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp 2024-03-11T18:10:13.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_async.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppoe.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppox.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pptp.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slhc.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slip.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/sungem_phy.ko 2024-03-11T18:10:13.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tap.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_random.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tun.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/veth.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/virtio_net.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan 2024-03-11T18:10:13.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan/vxlan.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netfront.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb/ntb.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme-core.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_cs.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_pc.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_serial.ko 2024-03-11T18:10:13.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/xen-pcifront.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/i82092.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/pd6729.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-11T18:10:13.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acerhdf.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/hdaps.ko 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel 2024-03-11T18:10:13.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel_ips.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-11T18:10:13.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps_parport.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/pps_core.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp_kvm.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-11T18:10:13.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-em3027.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-11T18:10:13.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-max6900.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-x1205.ko 2024-03-11T18:10:13.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-9xxx.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-sas.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-xxxx.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/BusLogic.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/a100u2w.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/advansys.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/am53c974.ko 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr 2024-03-11T18:10:13.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/atp870u.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa/bfa.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ch.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-11T18:10:13.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dc395x.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-11T18:10:13.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dmx3191d.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esp_scsi.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic/fnic.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hpsa.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hptiop.ko 2024-03-11T18:10:13.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hv_storvsc.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/initio.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ipr.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ips.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci/isci.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc/libfc.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas/libsas.ko 2024-03-11T18:10:13.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvumi.ko 2024-03-11T18:10:13.367 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pmcraid.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla1280.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/raid_class.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_debug.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-11T18:10:13.368 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ses.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/st.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/stex.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/virtio_scsi.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/wd719x.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb/ssb.ko 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi 2024-03-11T18:10:13.369 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_file.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_iblock.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_mod.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_pscsi.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_user.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-11T18:10:13.370 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-11T18:10:13.371 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_gsm.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_hdlc.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/nozomi.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/arc_uart.ko 2024-03-11T18:10:13.372 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_aec.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_cif.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_pci_generic.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_sercos3.ko 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb 2024-03-11T18:10:13.373 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-acm.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usblp.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usbtmc.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/mdc800.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/microtek.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/adutux.ko 2024-03-11T18:10:13.374 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/appledisplay.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi26.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi62.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ezusb.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/idmouse.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/iowarrior.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ldusb.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/legousbtower.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usb3503.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usblcd.ko 2024-03-11T18:10:13.375 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/uss720.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/yurex.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy/phy-generic.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/aircable.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ark3116.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ch341.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cp210x.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cyberjack.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-11T18:10:13.376 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/empeg.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_ti.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipaq.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipw.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ir-usb.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-11T18:10:13.377 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mct_u232.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7720.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7840.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/navman.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/omninet.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/opticon.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/option.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/oti6858.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/pl2303.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcaux.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcserial.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/quatech2.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/safe_serial.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/sierra.ko 2024-03-11T18:10:13.378 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ssu100.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/symbolserial.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_debug.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/visor.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/whiteheat.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/uas.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-11T18:10:13.379 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-11T18:10:13.381 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-11T18:10:13.381 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-karma.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/usb-storage.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio 2024-03-11T18:10:13.382 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_net.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_scsi.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/apple_bl.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/lcd.ko 2024-03-11T18:10:13.383 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_balloon.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_mmio.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_ring.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2482.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2490.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-11T18:10:13.384 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/wire.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-11T18:10:13.385 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/hpwdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/i6300esb.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ib700wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ibmasr.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it87_wdt.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/machzwd.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/nv_tco.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-11T18:10:13.386 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/softdog.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/via_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/wdt_pci.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/xen_wdt.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-evtchn.ko 2024-03-11T18:10:13.387 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntalloc.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntdev.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-privcmd.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p/9p.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs/affs.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs/befs.ko 2024-03-11T18:10:13.388 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/binfmt_misc.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs/btrfs.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles/cachefiles.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph/ceph.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda/coda.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs/cramfs.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm/dlm.ko 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs 2024-03-11T18:10:13.389 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs/f2fs.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/fat.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/msdos.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/vfat.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/cuse.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/fuse.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2/gfs2.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs/hfs.ko 2024-03-11T18:10:13.390 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus/hfsplus.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs/isofs.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs/jfs.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd/lockd.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix/minix.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs/netfs.ko 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout 2024-03-11T18:10:13.391 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfs.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv3.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv4.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/grace.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/nfs_acl.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd/nfsd.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2/nilfs2.ko 2024-03-11T18:10:13.392 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-celtic.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-centeuro.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-croatian.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-cyrillic.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-gaelic.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-greek.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-iceland.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-inuit.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-roman.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-romanian.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-turkish.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1250.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1251.ko 2024-03-11T18:10:13.393 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1255.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp737.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp775.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp850.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp852.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp855.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp857.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp860.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp861.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp862.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp863.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp864.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp865.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp866.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp869.ko 2024-03-11T18:10:13.394 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp874.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp932.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp936.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp949.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp950.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_euc-jp.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-1.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-13.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-14.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-15.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-2.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-3.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-4.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-5.ko 2024-03-11T18:10:13.395 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-6.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-7.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-9.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-r.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-ru.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-u.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_utf8.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-11T18:10:13.396 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs/overlay.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore/ramoops.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs/reiserfs.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs/romfs.ko 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb 2024-03-11T18:10:13.397 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client/cifs.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_arc4.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_md4.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs/squashfs.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv/sysv.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf/udf.ko 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs 2024-03-11T18:10:13.398 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs/ufs.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs/xfs.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu/rcutorture.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/torture.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/asn1_encoder.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc-itu-t.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc8.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libarc4.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libdes.ko 2024-03-11T18:10:13.399 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/libcrc32c.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lru_cache.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4_compress.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4hc_compress.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math/cordic.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6/raid6_pq.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_bm.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_fsm.ko 2024-03-11T18:10:13.400 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_kmp.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd/zstd_compress.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm/hwpoison-inject.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/6lowpan.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_dest.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_fragment.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_hop.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_mobility.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_routing.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_udp.ko 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802 2024-03-11T18:10:13.401 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/garp.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/mrp.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/p8022.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/psnap.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/stp.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q/8021q.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_fd.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_rdma.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_virtio.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv/batman-adv.ko 2024-03-11T18:10:13.402 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/br_netfilter.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/bridge.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-11T18:10:13.403 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtables.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-11T18:10:13.404 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph/libceph.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/failover.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/pktgen.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_diag.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv4.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv6.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver/dns_resolver.ko 2024-03-11T18:10:13.405 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/dsa_core.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_brcm.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_dsa.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_none.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_trailer.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ah4.ko 2024-03-11T18:10:13.406 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/esp4.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/fou.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/gre.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/inet_diag.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_gre.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_tunnel.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_vti.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipcomp.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipip.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-11T18:10:13.407 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-11T18:10:13.408 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_bic.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_dctcp.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_diag.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_highspeed.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_htcp.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_hybla.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_illinois.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_lp.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_scalable.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_vegas.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_veno.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_westwood.ko 2024-03-11T18:10:13.409 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_yeah.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tunnel4.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_diag.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_tunnel.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ah6.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/esp6.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/fou6.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_tunnel.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_vti.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ipcomp6.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-11T18:10:13.410 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-11T18:10:13.411 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/sit.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/tunnel6.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key/af_key.ko 2024-03-11T18:10:13.412 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_core.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_eth.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip6.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_netlink.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ppp.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc/llc.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154/mac802154.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls/mpls_gso.ko 2024-03-11T18:10:13.413 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-11T18:10:13.414 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-11T18:10:13.415 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conncount.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-11T18:10:13.416 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_log_syslog.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_irc.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_sip.ko 2024-03-11T18:10:13.417 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_tables.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_log.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_chain_nat.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_compat.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_ct.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fib.ko 2024-03-11T18:10:13.418 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_hash.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_limit.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_log.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_masq.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_nat.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_numgen.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_queue.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_quota.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_redir.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject_inet.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_socket.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_tunnel.ko 2024-03-11T18:10:13.419 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_AUDIT.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CT.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_DSCP.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HL.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HMARK.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LED.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LOG.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NETMAP.ko 2024-03-11T18:10:13.420 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFLOG.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_RATEEST.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_SECMARK.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TEE.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TPROXY.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TRACE.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_addrtype.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_bpf.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cgroup.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cluster.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_comment.ko 2024-03-11T18:10:13.421 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connbytes.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlabel.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlimit.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connmark.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_conntrack.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cpu.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dccp.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_devgroup.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dscp.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ecn.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_esp.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hashlimit.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_helper.ko 2024-03-11T18:10:13.422 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hl.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipcomp.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_iprange.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipvs.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_l2tp.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_length.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_limit.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mac.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mark.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_multiport.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nat.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nfacct.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_osf.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_owner.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_physdev.ko 2024-03-11T18:10:13.423 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_pkttype.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_policy.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_quota.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_rateest.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_realm.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_recent.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_sctp.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_set.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_socket.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_state.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_statistic.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_string.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_tcpmss.ko 2024-03-11T18:10:13.424 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_time.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_u32.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink/netlink_diag.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh/nsh.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/openvswitch.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-geneve.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-gre.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-vxlan.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet/af_packet_diag.ko 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds 2024-03-11T18:10:13.425 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_rdma.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_tcp.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_csum.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_gact.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_mirred.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_nat.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_pedit.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_police.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_simple.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_skbedit.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_vlan.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_basic.ko 2024-03-11T18:10:13.426 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_bpf.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_flow.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_fw.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_route.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_u32.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_cmp.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_ipset.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_meta.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_nbyte.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_text.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_u32.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_choke.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_codel.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_drr.ko 2024-03-11T18:10:13.427 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_fq.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_gred.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hfsc.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hhf.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_htb.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_ingress.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio_lib.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_multiq.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_netem.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_pie.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_plug.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_prio.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_qfq.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_red.ko 2024-03-11T18:10:13.428 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfb.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfq.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_tbf.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_teql.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp_diag.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/sunrpc.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-11T18:10:13.429 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/diag.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/tipc.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix/unix_diag.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-11T18:10:13.430 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys/trusted.ko 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib/irqbypass.ko 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias.bin 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.alias.bin 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.bin 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.modinfo 2024-03-11T18:10:13.431 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep.bin 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.devname 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.order 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.softdep 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols.bin 2024-03-11T18:10:13.432 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-11T18:10:13.436 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:13.436 DEBUG:teuthology.orchestra.run.smithi002:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-11T18:10:13.459 INFO:teuthology.orchestra.run.smithi002.stdout:postinstall scriptlet (using /bin/sh): 2024-03-11T18:10:13.459 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-11T18:10:13.459 INFO:teuthology.orchestra.run.smithi002.stdout: /usr/bin/kernel-install add 6.8.0-rc7-gdd5c1cd17167 /lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-11T18:10:13.459 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-03-11T18:10:13.459 INFO:teuthology.orchestra.run.smithi002.stdout:for file in vmlinuz System.map config; do 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167"; then 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: cp "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167" 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout: fi 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:done 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:new-kernel-pkg --remove 6.8.0-rc7-gdd5c1cd17167 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-gdd5c1cd17167.img 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-install remove 6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-11T18:10:13.460 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:13.461 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-03-11T18:10:13.461 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 68.el9 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T18:10:13.541 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8264254 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-03-11T18:10:13.542 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:10:13.543 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T18:10:13.544 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T18:10:13.544 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:13.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T18:10:14.161 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-03-11T18:10:15.519 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T18:10:15.551 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-03-11T18:10:15.553 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-03-11T18:10:15.568 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.usbAlf4BVM 2024-03-11T18:10:15.568 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.usbAlf4BVM 2024-03-11T18:10:15.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.usbAlf4BVM 2024-03-11T18:10:15.786 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.usbAlf4BVM is 6KB 2024-03-11T18:10:15.835 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.usbAlf4BVM 2024-03-11T18:10:15.850 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T18:10:15.926 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T18:10:15.926 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T18:10:15.926 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T18:10:15.926 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167.conf 2024-03-11T18:10:15.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:10:16.078 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-03-11T18:10:46.081 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-11T18:10:46.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T18:10:46.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:11:46.085 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T18:11:55.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T18:11:55.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:11:55.552 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-03-11T18:11:56.219 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T18:11:56.220 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-gdd5c1cd17167"... 2024-03-11T18:11:56.220 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-03-11T18:11:56.264 INFO:teuthology.orchestra.run.smithi002.stdout:6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:11:56.264 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-gdd5c1cd17167 vs 6.8.0-rc7-gdd5c1cd17167 2024-03-11T18:11:56.264 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T18:11:56.264 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T18:11:57.267 DEBUG:teuthology.parallel:result is None 2024-03-11T18:11:57.268 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T18:11:57.276 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T18:11:57.276 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:11:57.282 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:11:57.285 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T18:11:57.323 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T18:11:57.330 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T18:11:57.339 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T18:11:57.339 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:11:57.344 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:11:57.346 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T18:11:57.451 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T18:11:57.457 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T18:11:57.458 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:11:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:11:57.475 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T18:11:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:11:57.491 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:11:57.496 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:11:57.517 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T18:11:57.567 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.572 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.591 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.597 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.603 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.684 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T18:11:57.691 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T18:11:57.699 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T18:11:57.699 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:11:57.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:11:57.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T18:11:57.772 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T18:11:57.782 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T18:11:57.782 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:11:57.786 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:11:57.788 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T18:11:57.819 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:57.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:58.029 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.178 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:11:58.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:11:58.263 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:58.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:58.365 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.498 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:11:58.498 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:11:58.570 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:58.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T18:11:58.676 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T18:11:58.763 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:11:58.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T18:11:58.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-03-11T18:11:58.839 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-11T18:11:58.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-03-11T18:11:58.883 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:11:58.917 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:11:58.970 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:11:59.494 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T18:11:59.502 INFO:teuthology.task.internal:Starting timer... 2024-03-11T18:11:59.502 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T18:11:59.526 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T18:11:59.534 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T18:11:59.535 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-03-11T18:11:59.648 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-03-11T18:11:59.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-11T18:11:59.707 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-11T18:11:59.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-03-11T18:11:59.762 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-03-11T18:11:59.764 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T18:11:59.764 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-03-11T18:11:59.806 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-03-11T18:11:59.807 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-11T18:11:59.826 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-11T18:11:59.826 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-03-11T18:11:59.847 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-03-11T18:11:59.847 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-03-11T18:11:59.847 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:11:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:11:59.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:11:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:11:59.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:11:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:12:00.000 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T18:12:00.000 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:12:00.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:12:00.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-03-11T18:12:00.173 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T18:12:00.184 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-11T18:12:00.190 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-11T18:12:00.336 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T18:12:00.353 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T18:12:00.354 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi023.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T18:18:04.695 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-03-11T18:18:04.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T18:18:04.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:18:04.850 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-03-11T18:18:05.007 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-03-11T18:18:05.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T18:18:05.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:18:05.125 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-11T18:18:05.195 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-11T18:18:05.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T18:18:05.196 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T18:18:05.286 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-03-11T18:18:05.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-03-11T18:18:05.355 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T18:18:05.367 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T18:18:05.367 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:18:05.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:18:05.375 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:18:05.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:18:05.378 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T18:18:05.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:18:05.416 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:18:05.427 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:18:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:18:05.445 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:18:05.475 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-03-11T18:18:05.476 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-11T18:18:05.479 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T18:18:05.489 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-03-11T18:18:05.494 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-11T18:18:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:18:05.514 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:18:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:18:05.534 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T18:18:05.534 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:18:05.566 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.570 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.582 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:18:05.585 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:18:05.585 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-11T18:18:05.586 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.586 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.586 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.586 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.630 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-03-11T18:18:05.675 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-03-11T18:18:05.730 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T18:18:05.784 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T18:18:05.920 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.929 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:18:05.931 INFO:teuthology.run_tasks:Running task install... 2024-03-11T18:18:05.940 DEBUG:teuthology.task.install:project ceph 2024-03-11T18:18:05.941 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-11T18:18:05.941 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T18:18:05.941 INFO:teuthology.task.install:Using flavor: default 2024-03-11T18:18:05.949 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T18:18:05.950 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-11T18:18:05.950 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:18:05.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:18:05.953 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:18:05.953 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:18:05.954 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T18:18:05.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:18:06.115 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:18:06.116 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:18:06.182 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:18:06.182 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:18:06.187 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T18:18:06.187 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T18:18:06.289 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:18:06.289 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:18:06.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:18:06.297 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:18:06.298 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:18:06.298 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:18:06.319 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:18:06.319 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:18:06.332 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:18:06.332 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:18:06.352 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T18:18:06.352 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:18:06.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T18:18:06.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:18:06.405 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:18:06.462 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-11T18:18:06.462 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T18:18:06.470 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:18:06.494 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:18:06.524 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-11T18:18:06.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-11T18:18:06.537 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-03-11T18:18:06.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-03-11T18:18:06.604 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T18:18:06.742 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T18:18:06.773 INFO:teuthology.orchestra.run.smithi023.stdout:74 files removed 2024-03-11T18:18:06.776 INFO:teuthology.orchestra.run.smithi165.stdout:74 files removed 2024-03-11T18:18:06.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:18:06.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:18:06.841 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2024-03-11T18:18:06.846 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2024-03-11T18:18:07.224 INFO:teuthology.orchestra.run.smithi002.stdout:74 files removed 2024-03-11T18:18:07.265 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-11T18:18:07.461 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 234 kB/s | 82 kB 00:00 2024-03-11T18:18:07.484 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 212 kB/s | 82 kB 00:00 2024-03-11T18:18:07.870 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-11T18:18:07.935 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-11T18:18:08.164 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 215 kB/s | 82 kB 00:00 2024-03-11T18:18:08.221 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-11T18:18:08.294 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-11T18:18:08.631 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-11T18:18:08.648 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-11T18:18:08.722 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-11T18:18:09.090 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-11T18:18:09.718 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T18:18:10.080 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-11T18:18:10.090 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-11T18:18:11.445 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-11T18:18:11.917 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-11T18:18:16.032 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-03-11T18:18:18.094 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:07 2024-03-11T18:18:19.314 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-11T18:18:21.922 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-11T18:18:23.682 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-11T18:18:25.282 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-11T18:18:26.343 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-11T18:18:27.357 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-11T18:18:29.150 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-11T18:18:30.949 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-11T18:18:31.485 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-11T18:18:34.046 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:34.047 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:34.059 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:18:34.060 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:18:34.061 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:18:34.062 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:34.064 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:34.066 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:18:34.067 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:18:34.152 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:18:34.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:18:34.177 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:18:34.178 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:18:34.179 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:18:34.180 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-11T18:18:34.181 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:18:34.182 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:18:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:18:34.184 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:18:34.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:18:34.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:18:34.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:18:34.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:18:34.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:18:34.190 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout:Install 150 Packages 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 24 Packages 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:18:34.191 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 223 M 2024-03-11T18:18:34.192 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-11T18:18:34.827 INFO:teuthology.orchestra.run.smithi023.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-11T18:18:35.011 INFO:teuthology.orchestra.run.smithi023.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 4.7 MB/s | 886 kB 00:00 2024-03-11T18:18:35.069 INFO:teuthology.orchestra.run.smithi023.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.4 MB/s | 145 kB 00:00 2024-03-11T18:18:35.370 INFO:teuthology.orchestra.run.smithi023.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 7.1 MB/s | 2.1 MB 00:00 2024-03-11T18:18:35.537 INFO:teuthology.orchestra.run.smithi023.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 8.9 MB/s | 1.5 MB 00:00 2024-03-11T18:18:35.612 INFO:teuthology.orchestra.run.smithi023.stdout:(6/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 5.5 MB/s | 5.3 MB 00:00 2024-03-11T18:18:35.971 INFO:teuthology.orchestra.run.smithi023.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-11T18:18:36.979 INFO:teuthology.orchestra.run.smithi023.stdout:(8/174): ceph-radosgw-19.0.0-1607.g94a53970.el9 9.7 MB/s | 9.8 MB 00:01 2024-03-11T18:18:37.026 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T18:18:37.037 INFO:teuthology.orchestra.run.smithi023.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 433 kB/s | 25 kB 00:00 2024-03-11T18:18:37.304 INFO:teuthology.orchestra.run.smithi023.stdout:(10/174): ceph-osd-19.0.0-1607.g94a53970.el9.x8 9.9 MB/s | 17 MB 00:01 2024-03-11T18:18:37.538 INFO:teuthology.orchestra.run.smithi023.stdout:(11/174): ceph-common-19.0.0-1607.g94a53970.el9 6.9 MB/s | 20 MB 00:02 2024-03-11T18:18:37.573 INFO:teuthology.orchestra.run.smithi023.stdout:(12/174): cephfs-mirror-19.0.0-1607.g94a53970.e 814 kB/s | 217 kB 00:00 2024-03-11T18:18:37.590 INFO:teuthology.orchestra.run.smithi023.stdout:(13/174): cephfs-shell-19.0.0-1607.g94a53970.el 539 kB/s | 27 kB 00:00 2024-03-11T18:18:37.622 INFO:teuthology.orchestra.run.smithi023.stdout:(14/174): libcephfs-devel-19.0.0-1607.g94a53970 621 kB/s | 31 kB 00:00 2024-03-11T18:18:37.697 INFO:teuthology.orchestra.run.smithi023.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.1 MB/s | 163 kB 00:00 2024-03-11T18:18:37.723 INFO:teuthology.orchestra.run.smithi023.stdout:(16/174): libcephfs2-19.0.0-1607.g94a53970.el9. 5.4 MB/s | 742 kB 00:00 2024-03-11T18:18:37.790 INFO:teuthology.orchestra.run.smithi023.stdout:(17/174): librados-devel-19.0.0-1607.g94a53970. 1.3 MB/s | 127 kB 00:00 2024-03-11T18:18:37.832 INFO:teuthology.orchestra.run.smithi023.stdout:(18/174): libradosstriper1-19.0.0-1607.g94a5397 4.5 MB/s | 498 kB 00:00 2024-03-11T18:18:37.916 INFO:teuthology.orchestra.run.smithi023.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 541 kB/s | 45 kB 00:00 2024-03-11T18:18:38.032 INFO:teuthology.orchestra.run.smithi023.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.1 MB/s | 131 kB 00:00 2024-03-11T18:18:38.133 INFO:teuthology.orchestra.run.smithi023.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 1.6 MB/s | 161 kB 00:00 2024-03-11T18:18:38.241 INFO:teuthology.orchestra.run.smithi023.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 2.9 MB/s | 320 kB 00:00 2024-03-11T18:18:38.334 INFO:teuthology.orchestra.run.smithi023.stdout:(23/174): librgw2-19.0.0-1607.g94a53970.el9.x86 9.4 MB/s | 5.1 MB 00:00 2024-03-11T18:18:38.367 INFO:teuthology.orchestra.run.smithi023.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 2.3 MB/s | 298 kB 00:00 2024-03-11T18:18:38.420 INFO:teuthology.orchestra.run.smithi023.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.1 MB/s | 99 kB 00:00 2024-03-11T18:18:38.453 INFO:teuthology.orchestra.run.smithi023.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 989 kB/s | 85 kB 00:00 2024-03-11T18:18:38.562 INFO:teuthology.orchestra.run.smithi023.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.5 MB/s | 171 kB 00:00 2024-03-11T18:18:38.654 INFO:teuthology.orchestra.run.smithi023.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 342 kB/s | 31 kB 00:00 2024-03-11T18:18:38.747 INFO:teuthology.orchestra.run.smithi023.stdout:(29/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 9.5 MB/s | 3.1 MB 00:00 2024-03-11T18:18:38.772 INFO:teuthology.orchestra.run.smithi023.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.2 MB/s | 138 kB 00:00 2024-03-11T18:18:38.964 INFO:teuthology.orchestra.run.smithi023.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 8.2 MB/s | 1.8 MB 00:00 2024-03-11T18:18:39.056 INFO:teuthology.orchestra.run.smithi023.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 2.6 MB/s | 247 kB 00:00 2024-03-11T18:18:39.147 INFO:teuthology.orchestra.run.smithi023.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 540 kB/s | 49 kB 00:00 2024-03-11T18:18:39.289 INFO:teuthology.orchestra.run.smithi023.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1607.g9 108 kB/s | 15 kB 00:00 2024-03-11T18:18:39.373 INFO:teuthology.orchestra.run.smithi023.stdout:(35/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.1 MB/s | 264 kB 00:00 2024-03-11T18:18:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:(36/174): cephadm-19.0.0-1607.g94a53970.el9.noa 6.7 MB/s | 748 kB 00:00 2024-03-11T18:18:39.557 INFO:teuthology.orchestra.run.smithi023.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 244 kB/s | 18 kB 00:00 2024-03-11T18:18:39.682 INFO:teuthology.orchestra.run.smithi023.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 5.2 MB/s | 662 kB 00:00 2024-03-11T18:18:39.774 INFO:teuthology.orchestra.run.smithi023.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 7.4 MB/s | 7.4 MB 00:01 2024-03-11T18:18:39.816 INFO:teuthology.orchestra.run.smithi023.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.7 MB/s | 920 kB 00:00 2024-03-11T18:18:39.820 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:39.822 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:39.842 INFO:teuthology.orchestra.run.smithi023.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-11T18:18:39.843 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:18:39.844 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:18:39.845 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:18:39.847 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:39.849 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:39.853 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:18:39.854 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:18:39.875 INFO:teuthology.orchestra.run.smithi023.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-11T18:18:39.900 INFO:teuthology.orchestra.run.smithi023.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 546 kB/s | 32 kB 00:00 2024-03-11T18:18:39.952 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:18:39.972 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:18:39.973 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:18:39.974 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:18:39.975 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:18:39.976 INFO:teuthology.orchestra.run.smithi002.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:18:39.977 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:18:39.978 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:18:39.979 INFO:teuthology.orchestra.run.smithi002.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:18:39.980 INFO:teuthology.orchestra.run.smithi002.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:18:39.981 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:18:39.982 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:18:39.983 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:18:39.984 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:18:39.985 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout:Installing weak dependencies: 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout:Install 150 Packages 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 24 Packages 2024-03-11T18:18:39.986 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:18:39.987 INFO:teuthology.orchestra.run.smithi023.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-11T18:18:39.987 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 223 M 2024-03-11T18:18:39.987 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-03-11T18:18:40.001 INFO:teuthology.orchestra.run.smithi023.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-03-11T18:18:40.026 INFO:teuthology.orchestra.run.smithi023.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-11T18:18:40.052 INFO:teuthology.orchestra.run.smithi023.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-11T18:18:40.077 INFO:teuthology.orchestra.run.smithi023.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-11T18:18:40.111 INFO:teuthology.orchestra.run.smithi023.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-11T18:18:40.144 INFO:teuthology.orchestra.run.smithi023.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-11T18:18:40.170 INFO:teuthology.orchestra.run.smithi023.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 804 kB/s | 46 kB 00:00 2024-03-11T18:18:40.195 INFO:teuthology.orchestra.run.smithi023.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-11T18:18:40.220 INFO:teuthology.orchestra.run.smithi023.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-11T18:18:40.246 INFO:teuthology.orchestra.run.smithi023.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-11T18:18:40.262 INFO:teuthology.orchestra.run.smithi023.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 823 kB/s | 34 kB 00:00 2024-03-11T18:18:40.688 INFO:teuthology.orchestra.run.smithi023.stdout:(56/174): ceph-test-19.0.0-1607.g94a53970.el9.x 13 MB/s | 47 MB 00:03 2024-03-11T18:18:40.772 INFO:teuthology.orchestra.run.smithi023.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 9.0 MB/s | 4.7 MB 00:00 2024-03-11T18:18:40.786 INFO:teuthology.orchestra.run.smithi002.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-11T18:18:40.813 INFO:teuthology.orchestra.run.smithi023.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.8 MB/s | 1.5 MB 00:00 2024-03-11T18:18:40.838 INFO:teuthology.orchestra.run.smithi023.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-11T18:18:40.864 INFO:teuthology.orchestra.run.smithi023.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-11T18:18:40.922 INFO:teuthology.orchestra.run.smithi023.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-11T18:18:40.986 INFO:teuthology.orchestra.run.smithi002.stdout:(2/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 4.3 MB/s | 886 kB 00:00 2024-03-11T18:18:41.031 INFO:teuthology.orchestra.run.smithi023.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-11T18:18:41.036 INFO:teuthology.orchestra.run.smithi002.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 2.9 MB/s | 145 kB 00:00 2024-03-11T18:18:41.056 INFO:teuthology.orchestra.run.smithi023.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 406 kB/s | 78 kB 00:00 2024-03-11T18:18:41.173 INFO:teuthology.orchestra.run.smithi023.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-11T18:18:41.195 INFO:teuthology.orchestra.run.smithi002.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 2.1 MB 00:00 2024-03-11T18:18:41.198 INFO:teuthology.orchestra.run.smithi023.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 149 kB/s | 24 kB 00:00 2024-03-11T18:18:41.224 INFO:teuthology.orchestra.run.smithi023.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 110 kB/s | 18 kB 00:00 2024-03-11T18:18:41.249 INFO:teuthology.orchestra.run.smithi023.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-11T18:18:41.274 INFO:teuthology.orchestra.run.smithi023.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-11T18:18:41.311 INFO:teuthology.orchestra.run.smithi002.stdout:(5/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-11T18:18:41.383 INFO:teuthology.orchestra.run.smithi023.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-11T18:18:41.410 INFO:teuthology.orchestra.run.smithi023.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-11T18:18:41.434 INFO:teuthology.orchestra.run.smithi023.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-11T18:18:41.458 INFO:teuthology.orchestra.run.smithi023.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-11T18:18:41.484 INFO:teuthology.orchestra.run.smithi023.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-11T18:18:41.517 INFO:teuthology.orchestra.run.smithi023.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-11T18:18:41.542 INFO:teuthology.orchestra.run.smithi023.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-11T18:18:41.577 INFO:teuthology.orchestra.run.smithi023.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-11T18:18:41.595 INFO:teuthology.orchestra.run.smithi002.stdout:(6/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 17 MB/s | 4.7 MB 00:00 2024-03-11T18:18:41.602 INFO:teuthology.orchestra.run.smithi023.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 566 kB/s | 47 kB 00:00 2024-03-11T18:18:41.627 INFO:teuthology.orchestra.run.smithi023.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 509 kB/s | 43 kB 00:00 2024-03-11T18:18:41.652 INFO:teuthology.orchestra.run.smithi023.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 778 kB/s | 39 kB 00:00 2024-03-11T18:18:41.911 INFO:teuthology.orchestra.run.smithi023.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 145 kB/s | 41 kB 00:00 2024-03-11T18:18:42.078 INFO:teuthology.orchestra.run.smithi023.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-11T18:18:42.103 INFO:teuthology.orchestra.run.smithi023.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 670 kB/s | 300 kB 00:00 2024-03-11T18:18:42.137 INFO:teuthology.orchestra.run.smithi023.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 253 kB/s | 56 kB 00:00 2024-03-11T18:18:42.162 INFO:teuthology.orchestra.run.smithi023.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-11T18:18:42.187 INFO:teuthology.orchestra.run.smithi023.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 527 kB/s | 44 kB 00:00 2024-03-11T18:18:42.212 INFO:teuthology.orchestra.run.smithi023.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-11T18:18:42.238 INFO:teuthology.orchestra.run.smithi023.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T18:18:42.272 INFO:teuthology.orchestra.run.smithi023.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-11T18:18:42.297 INFO:teuthology.orchestra.run.smithi023.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T18:18:42.322 INFO:teuthology.orchestra.run.smithi023.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-11T18:18:42.355 INFO:teuthology.orchestra.run.smithi023.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 187 kB 00:00 2024-03-11T18:18:42.472 INFO:teuthology.orchestra.run.smithi023.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-11T18:18:42.498 INFO:teuthology.orchestra.run.smithi023.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-11T18:18:42.503 INFO:teuthology.orchestra.run.smithi002.stdout:(7/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 18 MB/s | 17 MB 00:00 2024-03-11T18:18:42.523 INFO:teuthology.orchestra.run.smithi023.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-11T18:18:42.557 INFO:teuthology.orchestra.run.smithi023.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-03-11T18:18:42.601 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 822 kB/s | 19 MB 00:23 2024-03-11T18:18:42.629 INFO:teuthology.orchestra.run.smithi002.stdout:(8/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 2.6 MB/s | 5.3 MB 00:02 2024-03-11T18:18:42.640 INFO:teuthology.orchestra.run.smithi023.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 218 kB/s | 25 kB 00:00 2024-03-11T18:18:42.666 INFO:teuthology.orchestra.run.smithi023.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 128 kB/s | 21 kB 00:00 2024-03-11T18:18:42.704 INFO:teuthology.orchestra.run.smithi002.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 342 kB/s | 25 kB 00:00 2024-03-11T18:18:42.732 INFO:teuthology.orchestra.run.smithi023.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-11T18:18:42.758 INFO:teuthology.orchestra.run.smithi023.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-11T18:18:42.808 INFO:teuthology.orchestra.run.smithi023.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 699 kB/s | 35 kB 00:00 2024-03-11T18:18:42.833 INFO:teuthology.orchestra.run.smithi023.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-11T18:18:42.910 INFO:teuthology.orchestra.run.smithi023.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-03-11T18:18:42.934 INFO:teuthology.orchestra.run.smithi023.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 355 kB/s | 44 kB 00:00 2024-03-11T18:18:42.959 INFO:teuthology.orchestra.run.smithi023.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 747 kB/s | 94 kB 00:00 2024-03-11T18:18:43.054 INFO:teuthology.orchestra.run.smithi002.stdout:(10/174): ceph-radosgw-19.0.0-1607.g94a53970.el 18 MB/s | 9.8 MB 00:00 2024-03-11T18:18:43.114 INFO:teuthology.orchestra.run.smithi002.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 3.7 MB/s | 217 kB 00:00 2024-03-11T18:18:43.163 INFO:teuthology.orchestra.run.smithi002.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 553 kB/s | 27 kB 00:00 2024-03-11T18:18:43.213 INFO:teuthology.orchestra.run.smithi002.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 637 kB/s | 31 kB 00:00 2024-03-11T18:18:43.314 INFO:teuthology.orchestra.run.smithi002.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 7.3 MB/s | 742 kB 00:00 2024-03-11T18:18:43.381 INFO:teuthology.orchestra.run.smithi002.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 2.4 MB/s | 163 kB 00:00 2024-03-11T18:18:43.431 INFO:teuthology.orchestra.run.smithi002.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 2.5 MB/s | 127 kB 00:00 2024-03-11T18:18:43.515 INFO:teuthology.orchestra.run.smithi002.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 5.9 MB/s | 498 kB 00:00 2024-03-11T18:18:43.517 INFO:teuthology.orchestra.run.smithi023.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 982 kB/s | 548 kB 00:00 2024-03-11T18:18:43.576 INFO:teuthology.orchestra.run.smithi023.stdout:(106/174): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-03-11T18:18:43.718 INFO:teuthology.orchestra.run.smithi023.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-11T18:18:43.900 INFO:teuthology.orchestra.run.smithi023.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 808 kB/s | 309 kB 00:00 2024-03-11T18:18:43.915 INFO:teuthology.orchestra.run.smithi002.stdout:(18/174): librgw2-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 5.1 MB 00:00 2024-03-11T18:18:43.965 INFO:teuthology.orchestra.run.smithi002.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 926 kB/s | 45 kB 00:00 2024-03-11T18:18:44.032 INFO:teuthology.orchestra.run.smithi002.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 2.0 MB/s | 131 kB 00:00 2024-03-11T18:18:44.044 INFO:teuthology.orchestra.run.smithi023.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 181 kB/s | 26 kB 00:00 2024-03-11T18:18:44.107 INFO:teuthology.orchestra.run.smithi002.stdout:(21/174): python3-cephfs-19.0.0-1607.g94a53970. 2.1 MB/s | 161 kB 00:00 2024-03-11T18:18:44.191 INFO:teuthology.orchestra.run.smithi002.stdout:(22/174): python3-rados-19.0.0-1607.g94a53970.e 3.8 MB/s | 320 kB 00:00 2024-03-11T18:18:44.252 INFO:teuthology.orchestra.run.smithi023.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-03-11T18:18:44.266 INFO:teuthology.orchestra.run.smithi002.stdout:(23/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.9 MB/s | 298 kB 00:00 2024-03-11T18:18:44.333 INFO:teuthology.orchestra.run.smithi002.stdout:(24/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.5 MB/s | 99 kB 00:00 2024-03-11T18:18:44.392 INFO:teuthology.orchestra.run.smithi002.stdout:(25/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.5 MB/s | 85 kB 00:00 2024-03-11T18:18:44.428 INFO:teuthology.orchestra.run.smithi023.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 385 kB/s | 67 kB 00:00 2024-03-11T18:18:44.734 INFO:teuthology.orchestra.run.smithi002.stdout:(26/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 9.1 MB/s | 3.1 MB 00:00 2024-03-11T18:18:44.744 INFO:teuthology.orchestra.run.smithi023.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 477 kB/s | 151 kB 00:00 2024-03-11T18:18:44.809 INFO:teuthology.orchestra.run.smithi002.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 2.3 MB/s | 171 kB 00:00 2024-03-11T18:18:44.867 INFO:teuthology.orchestra.run.smithi002.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 546 kB/s | 31 kB 00:00 2024-03-11T18:18:44.870 INFO:teuthology.orchestra.run.smithi023.stdout:(113/174): protobuf-compiler-3.14.0-13.el9.x86_ 446 kB/s | 863 kB 00:01 2024-03-11T18:18:44.943 INFO:teuthology.orchestra.run.smithi002.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.8 MB/s | 138 kB 00:00 2024-03-11T18:18:45.151 INFO:teuthology.orchestra.run.smithi002.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 8.6 MB/s | 1.8 MB 00:00 2024-03-11T18:18:45.304 INFO:teuthology.orchestra.run.smithi023.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.8 MB/s | 4.4 MB 00:01 2024-03-11T18:18:45.429 INFO:teuthology.orchestra.run.smithi023.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 348 kB/s | 43 kB 00:00 2024-03-11T18:18:45.537 INFO:teuthology.orchestra.run.smithi023.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 821 kB/s | 548 kB 00:00 2024-03-11T18:18:45.571 INFO:teuthology.orchestra.run.smithi023.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.0 MB/s | 838 kB 00:00 2024-03-11T18:18:45.613 INFO:teuthology.orchestra.run.smithi023.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 175 kB/s | 32 kB 00:00 2024-03-11T18:18:45.663 INFO:teuthology.orchestra.run.smithi023.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-03-11T18:18:45.788 INFO:teuthology.orchestra.run.smithi023.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 792 kB/s | 172 kB 00:00 2024-03-11T18:18:45.835 INFO:teuthology.orchestra.run.smithi002.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-11T18:18:45.893 INFO:teuthology.orchestra.run.smithi002.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1607.g94 4.2 MB/s | 247 kB 00:00 2024-03-11T18:18:45.944 INFO:teuthology.orchestra.run.smithi002.stdout:(33/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 1.0 MB/s | 49 kB 00:00 2024-03-11T18:18:45.964 INFO:teuthology.orchestra.run.smithi023.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.0 MB/s | 358 kB 00:00 2024-03-11T18:18:45.989 INFO:teuthology.orchestra.run.smithi023.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 176 kB/s | 35 kB 00:00 2024-03-11T18:18:46.306 INFO:teuthology.orchestra.run.smithi023.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 472 kB/s | 303 kB 00:00 2024-03-11T18:18:46.415 INFO:teuthology.orchestra.run.smithi023.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 450 kB/s | 203 kB 00:00 2024-03-11T18:18:46.490 INFO:teuthology.orchestra.run.smithi023.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-11T18:18:46.561 INFO:teuthology.orchestra.run.smithi002.stdout:(34/174): ceph-test-19.0.0-1607.g94a53970.el9.x 12 MB/s | 47 MB 00:03 2024-03-11T18:18:46.586 INFO:teuthology.orchestra.run.smithi002.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1607.g9 24 kB/s | 15 kB 00:00 2024-03-11T18:18:46.598 INFO:teuthology.orchestra.run.smithi023.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-03-11T18:18:46.624 INFO:teuthology.orchestra.run.smithi023.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 455 kB/s | 144 kB 00:00 2024-03-11T18:18:46.636 INFO:teuthology.orchestra.run.smithi002.stdout:(36/174): ceph-volume-19.0.0-1607.g94a53970.el9 3.5 MB/s | 264 kB 00:00 2024-03-11T18:18:46.676 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-11T18:18:46.686 INFO:teuthology.orchestra.run.smithi002.stdout:(37/174): cephfs-top-19.0.0-1607.g94a53970.el9. 373 kB/s | 18 kB 00:00 2024-03-11T18:18:46.707 INFO:teuthology.orchestra.run.smithi023.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-11T18:18:46.733 INFO:teuthology.orchestra.run.smithi023.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-11T18:18:46.803 INFO:teuthology.orchestra.run.smithi002.stdout:(38/174): cephadm-19.0.0-1607.g94a53970.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-11T18:18:46.866 INFO:teuthology.orchestra.run.smithi023.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 124 kB/s | 20 kB 00:00 2024-03-11T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-11T18:18:46.962 INFO:teuthology.orchestra.run.smithi002.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.8 MB/s | 920 kB 00:00 2024-03-11T18:18:46.987 INFO:teuthology.orchestra.run.smithi002.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-11T18:18:47.021 INFO:teuthology.orchestra.run.smithi002.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-11T18:18:47.033 INFO:teuthology.orchestra.run.smithi023.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 278 kB/s | 46 kB 00:00 2024-03-11T18:18:47.046 INFO:teuthology.orchestra.run.smithi002.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 577 kB/s | 32 kB 00:00 2024-03-11T18:18:47.146 INFO:teuthology.orchestra.run.smithi002.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-11T18:18:47.172 INFO:teuthology.orchestra.run.smithi002.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 17 kB 00:00 2024-03-11T18:18:47.183 INFO:teuthology.orchestra.run.smithi023.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 526 kB/s | 79 kB 00:00 2024-03-11T18:18:47.197 INFO:teuthology.orchestra.run.smithi002.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 425 kB/s | 10 kB 00:00 2024-03-11T18:18:47.231 INFO:teuthology.orchestra.run.smithi002.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-11T18:18:47.248 INFO:teuthology.orchestra.run.smithi002.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 924 kB/s | 44 kB 00:00 2024-03-11T18:18:47.298 INFO:teuthology.orchestra.run.smithi002.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2024-03-11T18:18:47.340 INFO:teuthology.orchestra.run.smithi002.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 6.7 MB/s | 578 kB 00:00 2024-03-11T18:18:47.365 INFO:teuthology.orchestra.run.smithi002.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 765 kB/s | 46 kB 00:00 2024-03-11T18:18:47.375 INFO:teuthology.orchestra.run.smithi023.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 301 kB/s | 58 kB 00:00 2024-03-11T18:18:47.399 INFO:teuthology.orchestra.run.smithi002.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-11T18:18:47.432 INFO:teuthology.orchestra.run.smithi002.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-11T18:18:47.450 INFO:teuthology.orchestra.run.smithi023.stdout:(134/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.4 MB/s | 2.0 MB 00:01 2024-03-11T18:18:47.457 INFO:teuthology.orchestra.run.smithi002.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.8 MB/s | 217 kB 00:00 2024-03-11T18:18:47.483 INFO:teuthology.orchestra.run.smithi002.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 724 kB/s | 34 kB 00:00 2024-03-11T18:18:47.484 INFO:teuthology.orchestra.run.smithi023.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 1.4 MB/s | 1.0 MB 00:00 2024-03-11T18:18:47.576 INFO:teuthology.orchestra.run.smithi023.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-11T18:18:47.633 INFO:teuthology.orchestra.run.smithi002.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-11T18:18:47.643 INFO:teuthology.orchestra.run.smithi023.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 568 kB/s | 90 kB 00:00 2024-03-11T18:18:47.683 INFO:teuthology.orchestra.run.smithi002.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.3 MB/s | 1.5 MB 00:00 2024-03-11T18:18:47.725 INFO:teuthology.orchestra.run.smithi002.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.7 MB/s | 811 kB 00:00 2024-03-11T18:18:47.726 INFO:teuthology.orchestra.run.smithi023.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 188 kB/s | 28 kB 00:00 2024-03-11T18:18:47.750 INFO:teuthology.orchestra.run.smithi002.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-11T18:18:47.794 INFO:teuthology.orchestra.run.smithi023.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 205 kB/s | 31 kB 00:00 2024-03-11T18:18:47.827 INFO:teuthology.orchestra.run.smithi023.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 603 kB/s | 272 kB 00:00 2024-03-11T18:18:47.893 INFO:teuthology.orchestra.run.smithi002.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-11T18:18:47.934 INFO:teuthology.orchestra.run.smithi002.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-11T18:18:47.976 INFO:teuthology.orchestra.run.smithi002.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-11T18:18:48.002 INFO:teuthology.orchestra.run.smithi023.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-11T18:18:48.028 INFO:teuthology.orchestra.run.smithi023.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 178 kB/s | 36 kB 00:00 2024-03-11T18:18:48.053 INFO:teuthology.orchestra.run.smithi023.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 578 kB/s | 188 kB 00:00 2024-03-11T18:18:48.118 INFO:teuthology.orchestra.run.smithi002.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-11T18:18:48.143 INFO:teuthology.orchestra.run.smithi002.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 24 kB 00:00 2024-03-11T18:18:48.153 INFO:teuthology.orchestra.run.smithi023.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 338 kB/s | 51 kB 00:00 2024-03-11T18:18:48.185 INFO:teuthology.orchestra.run.smithi002.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 449 kB/s | 18 kB 00:00 2024-03-11T18:18:48.211 INFO:teuthology.orchestra.run.smithi002.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-11T18:18:48.236 INFO:teuthology.orchestra.run.smithi002.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 653 kB/s | 32 kB 00:00 2024-03-11T18:18:48.245 INFO:teuthology.orchestra.run.smithi023.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-03-11T18:18:48.271 INFO:teuthology.orchestra.run.smithi023.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 413 kB/s | 90 kB 00:00 2024-03-11T18:18:48.354 INFO:teuthology.orchestra.run.smithi023.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-11T18:18:48.370 INFO:teuthology.orchestra.run.smithi002.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-11T18:18:48.395 INFO:teuthology.orchestra.run.smithi002.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-11T18:18:48.435 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-03-11T18:18:48.445 INFO:teuthology.orchestra.run.smithi002.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.5 MB/s | 174 kB 00:00 2024-03-11T18:18:48.470 INFO:teuthology.orchestra.run.smithi002.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 934 kB/s | 69 kB 00:00 2024-03-11T18:18:48.480 INFO:teuthology.orchestra.run.smithi023.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.3 MB/s | 427 kB 00:00 2024-03-11T18:18:48.512 INFO:teuthology.orchestra.run.smithi002.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-03-11T18:18:48.554 INFO:teuthology.orchestra.run.smithi002.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-11T18:18:48.579 INFO:teuthology.orchestra.run.smithi002.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-11T18:18:48.613 INFO:teuthology.orchestra.run.smithi002.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 211 kB 00:00 2024-03-11T18:18:48.638 INFO:teuthology.orchestra.run.smithi002.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 846 kB/s | 47 kB 00:00 2024-03-11T18:18:48.655 INFO:teuthology.orchestra.run.smithi023.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 499 kB/s | 192 kB 00:00 2024-03-11T18:18:48.663 INFO:teuthology.orchestra.run.smithi002.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 872 kB/s | 43 kB 00:00 2024-03-11T18:18:49.039 INFO:teuthology.orchestra.run.smithi002.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-11T18:18:49.064 INFO:teuthology.orchestra.run.smithi002.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 100 kB/s | 39 kB 00:00 2024-03-11T18:18:49.089 INFO:teuthology.orchestra.run.smithi002.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 835 kB/s | 41 kB 00:00 2024-03-11T18:18:49.123 INFO:teuthology.orchestra.run.smithi002.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-11T18:18:49.148 INFO:teuthology.orchestra.run.smithi002.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-03-11T18:18:49.174 INFO:teuthology.orchestra.run.smithi002.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-11T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 906 kB/s | 44 kB 00:00 2024-03-11T18:18:49.216 INFO:teuthology.orchestra.run.smithi002.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-11T18:18:49.249 INFO:teuthology.orchestra.run.smithi002.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 241 kB 00:00 2024-03-11T18:18:49.300 INFO:teuthology.orchestra.run.smithi002.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-11T18:18:49.313 INFO:teuthology.orchestra.run.smithi023.stdout:(150/174): librados2-19.0.0-1607.g94a53970.el9. 4.0 MB/s | 3.4 MB 00:00 2024-03-11T18:18:49.325 INFO:teuthology.orchestra.run.smithi002.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-11T18:18:49.350 INFO:teuthology.orchestra.run.smithi002.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-11T18:18:49.364 INFO:teuthology.orchestra.run.smithi023.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-11T18:18:49.384 INFO:teuthology.orchestra.run.smithi002.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-11T18:18:49.451 INFO:teuthology.orchestra.run.smithi002.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-11T18:18:49.501 INFO:teuthology.orchestra.run.smithi002.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 803 kB 00:00 2024-03-11T18:18:49.551 INFO:teuthology.orchestra.run.smithi002.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-11T18:18:49.589 INFO:teuthology.orchestra.run.smithi023.stdout:(152/174): librbd1-19.0.0-1607.g94a53970.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-03-11T18:18:49.593 INFO:teuthology.orchestra.run.smithi002.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-11T18:18:49.839 INFO:teuthology.orchestra.run.smithi023.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.8 MB/s | 2.3 MB 00:00 2024-03-11T18:18:49.894 INFO:teuthology.orchestra.run.smithi002.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 63 kB/s | 21 kB 00:00 2024-03-11T18:18:49.898 INFO:teuthology.orchestra.run.smithi023.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 725 kB/s | 42 kB 00:00 2024-03-11T18:18:49.919 INFO:teuthology.orchestra.run.smithi002.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 78 kB/s | 25 kB 00:00 2024-03-11T18:18:49.948 INFO:teuthology.orchestra.run.smithi023.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.9 MB/s | 248 kB 00:00 2024-03-11T18:18:49.990 INFO:teuthology.orchestra.run.smithi023.stdout:(156/174): thrift-0.15.0-2.el9.x86_64.rpm 990 kB/s | 1.6 MB 00:01 2024-03-11T18:18:49.994 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-11T18:18:50.015 INFO:teuthology.orchestra.run.smithi023.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-11T18:18:50.069 INFO:teuthology.orchestra.run.smithi002.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-11T18:18:50.074 INFO:teuthology.orchestra.run.smithi023.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-11T18:18:50.099 INFO:teuthology.orchestra.run.smithi023.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 335 kB/s | 36 kB 00:00 2024-03-11T18:18:50.133 INFO:teuthology.orchestra.run.smithi023.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-11T18:18:50.152 INFO:teuthology.orchestra.run.smithi002.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-03-11T18:18:50.158 INFO:teuthology.orchestra.run.smithi023.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-11T18:18:50.183 INFO:teuthology.orchestra.run.smithi023.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-11T18:18:50.234 INFO:teuthology.orchestra.run.smithi023.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-03-11T18:18:50.259 INFO:teuthology.orchestra.run.smithi023.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-11T18:18:50.261 INFO:teuthology.orchestra.run.smithi002.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-03-11T18:18:50.284 INFO:teuthology.orchestra.run.smithi023.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-11T18:18:50.295 INFO:teuthology.orchestra.run.smithi002.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.5 MB/s | 216 kB 00:00 2024-03-11T18:18:50.310 INFO:teuthology.orchestra.run.smithi023.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 574 kB/s | 29 kB 00:00 2024-03-11T18:18:50.345 INFO:teuthology.orchestra.run.smithi002.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 422 kB/s | 35 kB 00:00 2024-03-11T18:18:50.370 INFO:teuthology.orchestra.run.smithi002.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 601 kB/s | 44 kB 00:00 2024-03-11T18:18:50.377 INFO:teuthology.orchestra.run.smithi023.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-11T18:18:50.421 INFO:teuthology.orchestra.run.smithi002.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-11T18:18:50.460 INFO:teuthology.orchestra.run.smithi023.stdout:(168/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-03-11T18:18:50.487 INFO:teuthology.orchestra.run.smithi002.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 193 kB/s | 22 kB 00:00 2024-03-11T18:18:50.519 INFO:teuthology.orchestra.run.smithi023.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 16 MB/s | 2.3 MB 00:00 2024-03-11T18:18:50.561 INFO:teuthology.orchestra.run.smithi023.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 750 kB/s | 31 kB 00:00 2024-03-11T18:18:50.597 INFO:teuthology.orchestra.run.smithi023.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 3.3 MB/s | 465 kB 00:00 2024-03-11T18:18:50.613 INFO:teuthology.orchestra.run.smithi002.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.3 MB/s | 548 kB 00:00 2024-03-11T18:18:50.630 INFO:teuthology.orchestra.run.smithi023.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-11T18:18:50.654 INFO:teuthology.orchestra.run.smithi002.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 7.4 MB/s | 309 kB 00:00 2024-03-11T18:18:50.680 INFO:teuthology.orchestra.run.smithi023.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 892 kB/s | 74 kB 00:00 2024-03-11T18:18:50.688 INFO:teuthology.orchestra.run.smithi002.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 597 kB/s | 19 kB 00:00 2024-03-11T18:18:50.772 INFO:teuthology.orchestra.run.smithi002.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-03-11T18:18:50.831 INFO:teuthology.orchestra.run.smithi002.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 455 kB/s | 26 kB 00:00 2024-03-11T18:18:50.923 INFO:teuthology.orchestra.run.smithi002.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-03-11T18:18:50.939 INFO:teuthology.orchestra.run.smithi023.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.4 MB/s | 2.2 MB 00:00 2024-03-11T18:18:50.943 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:18:50.943 INFO:teuthology.orchestra.run.smithi023.stdout:Total 13 MB/s | 223 MB 00:16 2024-03-11T18:18:50.948 INFO:teuthology.orchestra.run.smithi002.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-03-11T18:18:50.965 INFO:teuthology.orchestra.run.smithi002.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-11T18:18:51.007 INFO:teuthology.orchestra.run.smithi002.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-03-11T18:18:51.057 INFO:teuthology.orchestra.run.smithi002.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-03-11T18:18:51.099 INFO:teuthology.orchestra.run.smithi002.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 5.9 MB/s | 548 kB 00:00 2024-03-11T18:18:51.124 INFO:teuthology.orchestra.run.smithi002.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 660 kB/s | 43 kB 00:00 2024-03-11T18:18:51.158 INFO:teuthology.orchestra.run.smithi002.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 471 kB/s | 15 kB 00:00 2024-03-11T18:18:51.183 INFO:teuthology.orchestra.run.smithi002.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 562 kB/s | 32 kB 00:00 2024-03-11T18:18:51.208 INFO:teuthology.orchestra.run.smithi002.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-11T18:18:51.242 INFO:teuthology.orchestra.run.smithi002.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 6.1 MB/s | 358 kB 00:00 2024-03-11T18:18:51.275 INFO:teuthology.orchestra.run.smithi002.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-11T18:18:51.301 INFO:teuthology.orchestra.run.smithi002.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 614 kB/s | 35 kB 00:00 2024-03-11T18:18:51.334 INFO:teuthology.orchestra.run.smithi002.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-11T18:18:51.376 INFO:teuthology.orchestra.run.smithi002.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.5 MB/s | 144 kB 00:00 2024-03-11T18:18:51.417 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-11T18:18:51.426 INFO:teuthology.orchestra.run.smithi002.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 218 kB/s | 11 kB 00:00 2024-03-11T18:18:51.502 INFO:teuthology.orchestra.run.smithi002.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-03-11T18:18:51.527 INFO:teuthology.orchestra.run.smithi002.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 178 kB/s | 18 kB 00:00 2024-03-11T18:18:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 470 kB/s | 23 kB 00:00 2024-03-11T18:18:51.577 INFO:teuthology.orchestra.run.smithi002.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 396 kB/s | 19 kB 00:00 2024-03-11T18:18:51.604 INFO:teuthology.orchestra.run.smithi002.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 399 kB/s | 20 kB 00:00 2024-03-11T18:18:51.637 INFO:teuthology.orchestra.run.smithi002.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-11T18:18:51.687 INFO:teuthology.orchestra.run.smithi002.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.5 MB/s | 1.0 MB 00:00 2024-03-11T18:18:51.712 INFO:teuthology.orchestra.run.smithi002.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-11T18:18:51.713 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-11T18:18:51.713 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-11T18:18:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-11T18:18:51.780 INFO:teuthology.orchestra.run.smithi002.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-11T18:18:51.804 INFO:teuthology.orchestra.run.smithi002.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 249 kB/s | 16 kB 00:00 2024-03-11T18:18:52.055 INFO:teuthology.orchestra.run.smithi002.stdout:(137/174): ceph-common-19.0.0-1607.g94a53970.el 1.7 MB/s | 20 MB 00:11 2024-03-11T18:18:52.072 INFO:teuthology.orchestra.run.smithi002.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-03-11T18:18:52.097 INFO:teuthology.orchestra.run.smithi002.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 284 kB/s | 90 kB 00:00 2024-03-11T18:18:52.114 INFO:teuthology.orchestra.run.smithi002.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 534 kB/s | 31 kB 00:00 2024-03-11T18:18:52.131 INFO:teuthology.orchestra.run.smithi002.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-03-11T18:18:52.156 INFO:teuthology.orchestra.run.smithi002.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-11T18:18:52.207 INFO:teuthology.orchestra.run.smithi002.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 392 kB/s | 36 kB 00:00 2024-03-11T18:18:52.232 INFO:teuthology.orchestra.run.smithi002.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 511 kB/s | 51 kB 00:00 2024-03-11T18:18:52.265 INFO:teuthology.orchestra.run.smithi002.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-11T18:18:52.291 INFO:teuthology.orchestra.run.smithi002.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-11T18:18:52.316 INFO:teuthology.orchestra.run.smithi002.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 406 kB/s | 20 kB 00:00 2024-03-11T18:18:52.349 INFO:teuthology.orchestra.run.smithi002.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-11T18:18:52.383 INFO:teuthology.orchestra.run.smithi002.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-03-11T18:18:52.458 INFO:teuthology.orchestra.run.smithi002.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-11T18:18:52.500 INFO:teuthology.orchestra.run.smithi002.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-11T18:18:52.667 INFO:teuthology.orchestra.run.smithi002.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-11T18:18:52.809 INFO:teuthology.orchestra.run.smithi002.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-03-11T18:18:52.868 INFO:teuthology.orchestra.run.smithi002.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 740 kB/s | 42 kB 00:00 2024-03-11T18:18:52.952 INFO:teuthology.orchestra.run.smithi002.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.0 MB/s | 248 kB 00:00 2024-03-11T18:18:53.035 INFO:teuthology.orchestra.run.smithi002.stdout:(156/174): librados2-19.0.0-1607.g94a53970.el9. 4.9 MB/s | 3.4 MB 00:00 2024-03-11T18:18:53.061 INFO:teuthology.orchestra.run.smithi002.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 657 kB/s | 71 kB 00:00 2024-03-11T18:18:53.103 INFO:teuthology.orchestra.run.smithi002.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 546 kB/s | 36 kB 00:00 2024-03-11T18:18:53.153 INFO:teuthology.orchestra.run.smithi002.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-03-11T18:18:53.204 INFO:teuthology.orchestra.run.smithi002.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-03-11T18:18:53.229 INFO:teuthology.orchestra.run.smithi002.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 311 kB/s | 23 kB 00:00 2024-03-11T18:18:53.288 INFO:teuthology.orchestra.run.smithi002.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-11T18:18:53.342 INFO:teuthology.orchestra.run.smithi002.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-11T18:18:53.405 INFO:teuthology.orchestra.run.smithi002.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-11T18:18:53.430 INFO:teuthology.orchestra.run.smithi002.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 735 kB/s | 64 kB 00:00 2024-03-11T18:18:53.472 INFO:teuthology.orchestra.run.smithi002.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 436 kB/s | 29 kB 00:00 2024-03-11T18:18:53.517 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-11T18:18:53.518 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-11T18:18:53.574 INFO:teuthology.orchestra.run.smithi002.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 1.2 MB 00:00 2024-03-11T18:18:53.689 INFO:teuthology.orchestra.run.smithi002.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-11T18:18:53.798 INFO:teuthology.orchestra.run.smithi002.stdout:(169/174): librbd1-19.0.0-1607.g94a53970.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-03-11T18:18:53.882 INFO:teuthology.orchestra.run.smithi002.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.4 MB/s | 2.3 MB 00:00 2024-03-11T18:18:53.898 INFO:teuthology.orchestra.run.smithi002.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 315 kB/s | 31 kB 00:00 2024-03-11T18:18:53.999 INFO:teuthology.orchestra.run.smithi002.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 1.5 MB/s | 465 kB 00:00 2024-03-11T18:18:54.024 INFO:teuthology.orchestra.run.smithi002.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 598 kB/s | 74 kB 00:00 2024-03-11T18:18:54.108 INFO:teuthology.orchestra.run.smithi002.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-03-11T18:18:54.118 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:18:54.118 INFO:teuthology.orchestra.run.smithi002.stdout:Total 16 MB/s | 223 MB 00:14 2024-03-11T18:18:54.961 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-11T18:18:55.626 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-03-11T18:18:56.024 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-03-11T18:18:56.024 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-03-11T18:18:57.408 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:57.409 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-11T18:18:57.423 INFO:teuthology.orchestra.run.smithi165.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-11T18:18:57.425 INFO:teuthology.orchestra.run.smithi165.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-11T18:18:57.426 INFO:teuthology.orchestra.run.smithi165.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-11T18:18:57.427 INFO:teuthology.orchestra.run.smithi165.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:57.429 INFO:teuthology.orchestra.run.smithi165.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-11T18:18:57.432 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-11T18:18:57.433 INFO:teuthology.orchestra.run.smithi165.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-11T18:18:57.514 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-11T18:18:57.533 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T18:18:57.534 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 217 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell x86_64 2:19.0.0-1607.g94a53970.el9 ceph 27 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 18 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-11T18:18:57.535 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-11T18:18:57.536 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-11T18:18:57.537 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-11T18:18:57.538 INFO:teuthology.orchestra.run.smithi165.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T18:18:57.539 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T18:18:57.540 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T18:18:57.541 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T18:18:57.542 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T18:18:57.543 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T18:18:57.544 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T18:18:57.545 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T18:18:57.546 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-03-11T18:18:57.547 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================================ 2024-03-11T18:18:57.548 INFO:teuthology.orchestra.run.smithi165.stdout:Install 150 Packages 2024-03-11T18:18:57.548 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 24 Packages 2024-03-11T18:18:57.548 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:18:57.548 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 223 M 2024-03-11T18:18:57.548 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-03-11T18:18:58.420 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-03-11T18:18:58.421 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-03-11T18:18:58.562 INFO:teuthology.orchestra.run.smithi165.stdout:(1/174): ceph-19.0.0-1607.g94a53970.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-11T18:18:58.786 INFO:teuthology.orchestra.run.smithi165.stdout:(2/174): ceph-base-19.0.0-1607.g94a53970.el9.x8 9.9 MB/s | 5.3 MB 00:00 2024-03-11T18:18:58.829 INFO:teuthology.orchestra.run.smithi165.stdout:(3/174): ceph-immutable-object-cache-19.0.0-160 3.4 MB/s | 145 kB 00:00 2024-03-11T18:18:58.995 INFO:teuthology.orchestra.run.smithi165.stdout:(4/174): ceph-mds-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 2.1 MB 00:00 2024-03-11T18:18:59.021 INFO:teuthology.orchestra.run.smithi165.stdout:(5/174): ceph-fuse-19.0.0-1607.g94a53970.el9.x8 1.9 MB/s | 886 kB 00:00 2024-03-11T18:18:59.138 INFO:teuthology.orchestra.run.smithi165.stdout:(6/174): ceph-mgr-19.0.0-1607.g94a53970.el9.x86 10 MB/s | 1.5 MB 00:00 2024-03-11T18:18:59.930 INFO:teuthology.orchestra.run.smithi165.stdout:(7/174): ceph-mon-19.0.0-1607.g94a53970.el9.x86 5.2 MB/s | 4.7 MB 00:00 2024-03-11T18:19:00.463 INFO:teuthology.orchestra.run.smithi165.stdout:(8/174): ceph-osd-19.0.0-1607.g94a53970.el9.x86 13 MB/s | 17 MB 00:01 2024-03-11T18:19:00.513 INFO:teuthology.orchestra.run.smithi165.stdout:(9/174): ceph-selinux-19.0.0-1607.g94a53970.el9 507 kB/s | 25 kB 00:00 2024-03-11T18:19:00.840 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-11T18:19:00.968 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:19:01.002 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:19:01.032 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:01.042 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:01.082 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:19:01.136 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:19:01.170 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:19:01.172 INFO:teuthology.orchestra.run.smithi165.stdout:(10/174): ceph-common-19.0.0-1607.g94a53970.el9 6.8 MB/s | 20 MB 00:02 2024-03-11T18:19:01.195 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:19:01.237 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:19:01.272 INFO:teuthology.orchestra.run.smithi165.stdout:(11/174): cephfs-mirror-19.0.0-1607.g94a53970.e 2.1 MB/s | 217 kB 00:00 2024-03-11T18:19:01.347 INFO:teuthology.orchestra.run.smithi165.stdout:(12/174): cephfs-shell-19.0.0-1607.g94a53970.el 361 kB/s | 27 kB 00:00 2024-03-11T18:19:01.423 INFO:teuthology.orchestra.run.smithi165.stdout:(13/174): libcephfs-devel-19.0.0-1607.g94a53970 415 kB/s | 31 kB 00:00 2024-03-11T18:19:01.573 INFO:teuthology.orchestra.run.smithi165.stdout:(14/174): libcephfs2-19.0.0-1607.g94a53970.el9. 4.8 MB/s | 742 kB 00:00 2024-03-11T18:19:01.630 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:19:01.656 INFO:teuthology.orchestra.run.smithi165.stdout:(15/174): libcephsqlite-19.0.0-1607.g94a53970.e 1.9 MB/s | 163 kB 00:00 2024-03-11T18:19:01.671 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:01.740 INFO:teuthology.orchestra.run.smithi165.stdout:(16/174): librados-devel-19.0.0-1607.g94a53970. 1.5 MB/s | 127 kB 00:00 2024-03-11T18:19:01.778 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:01.840 INFO:teuthology.orchestra.run.smithi165.stdout:(17/174): libradosstriper1-19.0.0-1607.g94a5397 4.9 MB/s | 498 kB 00:00 2024-03-11T18:19:01.881 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:01.909 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:19:01.949 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:19:02.135 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:19:02.174 INFO:teuthology.orchestra.run.smithi165.stdout:(18/174): ceph-radosgw-19.0.0-1607.g94a53970.el 4.3 MB/s | 9.8 MB 00:02 2024-03-11T18:19:02.182 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:19:02.210 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:02.266 INFO:teuthology.orchestra.run.smithi165.stdout:(19/174): python3-ceph-argparse-19.0.0-1607.g94 493 kB/s | 45 kB 00:00 2024-03-11T18:19:02.314 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:02.336 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:02.357 INFO:teuthology.orchestra.run.smithi165.stdout:(20/174): python3-ceph-common-19.0.0-1607.g94a5 1.4 MB/s | 131 kB 00:00 2024-03-11T18:19:02.400 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:02.440 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:19:02.449 INFO:teuthology.orchestra.run.smithi165.stdout:(21/174): librgw2-19.0.0-1607.g94a53970.el9.x86 8.4 MB/s | 5.1 MB 00:00 2024-03-11T18:19:02.463 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:02.475 INFO:teuthology.orchestra.run.smithi165.stdout:(22/174): python3-cephfs-19.0.0-1607.g94a53970. 1.3 MB/s | 161 kB 00:00 2024-03-11T18:19:02.478 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:02.542 INFO:teuthology.orchestra.run.smithi165.stdout:(23/174): python3-rados-19.0.0-1607.g94a53970.e 3.4 MB/s | 320 kB 00:00 2024-03-11T18:19:02.567 INFO:teuthology.orchestra.run.smithi165.stdout:(24/174): python3-rbd-19.0.0-1607.g94a53970.el9 3.2 MB/s | 298 kB 00:00 2024-03-11T18:19:02.617 INFO:teuthology.orchestra.run.smithi165.stdout:(25/174): python3-rgw-19.0.0-1607.g94a53970.el9 1.3 MB/s | 99 kB 00:00 2024-03-11T18:19:02.642 INFO:teuthology.orchestra.run.smithi165.stdout:(26/174): rbd-fuse-19.0.0-1607.g94a53970.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-11T18:19:02.646 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:02.677 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:02.708 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:19:02.743 INFO:teuthology.orchestra.run.smithi165.stdout:(27/174): rbd-nbd-19.0.0-1607.g94a53970.el9.x86 1.7 MB/s | 171 kB 00:00 2024-03-11T18:19:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:19:02.826 INFO:teuthology.orchestra.run.smithi165.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1607.g 377 kB/s | 31 kB 00:00 2024-03-11T18:19:02.919 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:02.927 INFO:teuthology.orchestra.run.smithi165.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1607.g94a5397 1.4 MB/s | 138 kB 00:00 2024-03-11T18:19:02.979 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:02.985 INFO:teuthology.orchestra.run.smithi165.stdout:(30/174): rbd-mirror-19.0.0-1607.g94a53970.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-11T18:19:03.052 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:03.097 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:03.294 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:03.352 INFO:teuthology.orchestra.run.smithi165.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1607.g94a53 4.2 MB/s | 1.8 MB 00:00 2024-03-11T18:19:03.422 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:19:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:19:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:19:03.612 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:19:03.648 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:19:03.725 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:19:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:19:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:19:03.853 INFO:teuthology.orchestra.run.smithi165.stdout:(32/174): ceph-test-19.0.0-1607.g94a53970.el9.x 14 MB/s | 47 MB 00:03 2024-03-11T18:19:03.878 INFO:teuthology.orchestra.run.smithi165.stdout:(33/174): ceph-mgr-modules-core-19.0.0-1607.g94 471 kB/s | 247 kB 00:00 2024-03-11T18:19:03.903 INFO:teuthology.orchestra.run.smithi165.stdout:(34/174): ceph-mgr-rook-19.0.0-1607.g94a53970.e 989 kB/s | 49 kB 00:00 2024-03-11T18:19:03.917 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:19:03.955 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:19:03.999 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:19:04.003 INFO:teuthology.orchestra.run.smithi165.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-03-11T18:19:04.028 INFO:teuthology.orchestra.run.smithi165.stdout:(36/174): ceph-prometheus-alerts-19.0.0-1607.g9 102 kB/s | 15 kB 00:00 2024-03-11T18:19:04.039 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:19:04.054 INFO:teuthology.orchestra.run.smithi165.stdout:(37/174): ceph-volume-19.0.0-1607.g94a53970.el9 1.7 MB/s | 264 kB 00:00 2024-03-11T18:19:04.076 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:19:04.112 INFO:teuthology.orchestra.run.smithi165.stdout:(38/174): cephfs-top-19.0.0-1607.g94a53970.el9. 219 kB/s | 18 kB 00:00 2024-03-11T18:19:04.148 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:19:04.180 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:19:04.196 INFO:teuthology.orchestra.run.smithi165.stdout:(39/174): cephadm-19.0.0-1607.g94a53970.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-11T18:19:04.224 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:19:04.229 INFO:teuthology.orchestra.run.smithi165.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.8 MB/s | 920 kB 00:00 2024-03-11T18:19:04.253 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:19:04.263 INFO:teuthology.orchestra.run.smithi165.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-11T18:19:04.288 INFO:teuthology.orchestra.run.smithi165.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-11T18:19:04.320 INFO:teuthology.orchestra.run.smithi165.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 307 kB 00:00 2024-03-11T18:19:04.345 INFO:teuthology.orchestra.run.smithi165.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 564 kB/s | 32 kB 00:00 2024-03-11T18:19:04.353 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:19:04.402 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:19:04.412 INFO:teuthology.orchestra.run.smithi165.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-11T18:19:04.437 INFO:teuthology.orchestra.run.smithi165.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 17 kB 00:00 2024-03-11T18:19:04.462 INFO:teuthology.orchestra.run.smithi165.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-03-11T18:19:04.485 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:19:04.488 INFO:teuthology.orchestra.run.smithi165.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-11T18:19:04.513 INFO:teuthology.orchestra.run.smithi165.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-11T18:19:04.539 INFO:teuthology.orchestra.run.smithi165.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-11T18:19:04.564 INFO:teuthology.orchestra.run.smithi165.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.7 MB/s | 578 kB 00:00 2024-03-11T18:19:04.589 INFO:teuthology.orchestra.run.smithi165.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 46 kB 00:00 2024-03-11T18:19:04.622 INFO:teuthology.orchestra.run.smithi165.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-11T18:19:04.623 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:19:04.648 INFO:teuthology.orchestra.run.smithi165.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-11T18:19:04.673 INFO:teuthology.orchestra.run.smithi165.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 689 kB/s | 34 kB 00:00 2024-03-11T18:19:04.688 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:19:04.698 INFO:teuthology.orchestra.run.smithi165.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-11T18:19:04.774 INFO:teuthology.orchestra.run.smithi165.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-11T18:19:04.816 INFO:teuthology.orchestra.run.smithi165.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-11T18:19:04.841 INFO:teuthology.orchestra.run.smithi165.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-11T18:19:04.866 INFO:teuthology.orchestra.run.smithi165.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-11T18:19:04.900 INFO:teuthology.orchestra.run.smithi165.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-11T18:19:04.942 INFO:teuthology.orchestra.run.smithi165.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 78 kB 00:00 2024-03-11T18:19:05.017 INFO:teuthology.orchestra.run.smithi165.stdout:(63/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-11T18:19:05.101 INFO:teuthology.orchestra.run.smithi165.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-11T18:19:05.126 INFO:teuthology.orchestra.run.smithi165.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 135 kB/s | 24 kB 00:00 2024-03-11T18:19:05.142 INFO:teuthology.orchestra.run.smithi165.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 147 kB/s | 18 kB 00:00 2024-03-11T18:19:05.177 INFO:teuthology.orchestra.run.smithi165.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-11T18:19:05.201 INFO:teuthology.orchestra.run.smithi165.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 544 kB/s | 32 kB 00:00 2024-03-11T18:19:05.253 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:19:05.268 INFO:teuthology.orchestra.run.smithi165.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-11T18:19:05.285 INFO:teuthology.orchestra.run.smithi165.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-11T18:19:05.310 INFO:teuthology.orchestra.run.smithi165.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 174 kB 00:00 2024-03-11T18:19:05.335 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:19:05.336 INFO:teuthology.orchestra.run.smithi165.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-11T18:19:05.361 INFO:teuthology.orchestra.run.smithi165.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-11T18:19:05.378 INFO:teuthology.orchestra.run.smithi165.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 615 kB/s | 41 kB 00:00 2024-03-11T18:19:05.403 INFO:teuthology.orchestra.run.smithi165.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-11T18:19:05.433 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:19:05.445 INFO:teuthology.orchestra.run.smithi165.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-11T18:19:05.470 INFO:teuthology.orchestra.run.smithi165.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 714 kB/s | 47 kB 00:00 2024-03-11T18:19:05.493 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:05.495 INFO:teuthology.orchestra.run.smithi165.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 468 kB/s | 43 kB 00:00 2024-03-11T18:19:05.501 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:05.541 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:19:05.582 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:19:05.620 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:19:05.649 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:19:05.774 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-03-11T18:19:05.813 INFO:teuthology.orchestra.run.smithi165.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-11T18:19:05.837 INFO:teuthology.orchestra.run.smithi165.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 114 kB/s | 39 kB 00:00 2024-03-11T18:19:05.856 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:19:05.902 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:19:05.912 INFO:teuthology.orchestra.run.smithi165.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 99 kB/s | 41 kB 00:00 2024-03-11T18:19:05.913 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:19:05.946 INFO:teuthology.orchestra.run.smithi165.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 300 kB 00:00 2024-03-11T18:19:05.952 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:05.971 INFO:teuthology.orchestra.run.smithi165.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-11T18:19:05.996 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:05.996 INFO:teuthology.orchestra.run.smithi165.stdout:(84/174): uuid-1.6.2-55.el9.x86_64.rpm 675 kB/s | 56 kB 00:00 2024-03-11T18:19:06.021 INFO:teuthology.orchestra.run.smithi165.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-11T18:19:06.047 INFO:teuthology.orchestra.run.smithi165.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-11T18:19:06.059 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:19:06.081 INFO:teuthology.orchestra.run.smithi165.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-11T18:19:06.088 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:19:06.122 INFO:teuthology.orchestra.run.smithi165.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-11T18:19:06.127 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:19:06.147 INFO:teuthology.orchestra.run.smithi165.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-11T18:19:06.171 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:19:06.173 INFO:teuthology.orchestra.run.smithi165.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-11T18:19:06.175 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:19:06.198 INFO:teuthology.orchestra.run.smithi165.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 187 kB 00:00 2024-03-11T18:19:06.222 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:19:06.257 INFO:teuthology.orchestra.run.smithi165.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-11T18:19:06.266 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:19:06.282 INFO:teuthology.orchestra.run.smithi165.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-11T18:19:06.312 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:19:06.315 INFO:teuthology.orchestra.run.smithi165.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-03-11T18:19:06.349 INFO:teuthology.orchestra.run.smithi165.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-11T18:19:06.441 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:19:06.556 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:19:06.608 INFO:teuthology.orchestra.run.smithi165.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 87 kB/s | 25 kB 00:00 2024-03-11T18:19:06.616 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:19:06.633 INFO:teuthology.orchestra.run.smithi165.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 61 kB/s | 21 kB 00:00 2024-03-11T18:19:06.688 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:19:06.725 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:19:06.761 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:19:06.809 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:19:06.844 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:19:06.888 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:19:06.930 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:19:06.933 INFO:teuthology.orchestra.run.smithi165.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 754 kB/s | 245 kB 00:00 2024-03-11T18:19:06.959 INFO:teuthology.orchestra.run.smithi165.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 762 kB/s | 248 kB 00:00 2024-03-11T18:19:06.989 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:19:07.023 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:07.025 INFO:teuthology.orchestra.run.smithi165.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 522 kB/s | 35 kB 00:00 2024-03-11T18:19:07.078 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:19:07.117 INFO:teuthology.orchestra.run.smithi165.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-03-11T18:19:07.151 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:19:07.182 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:19:07.215 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:07.223 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:19:07.249 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:07.261 INFO:teuthology.orchestra.run.smithi165.stdout:(102/174): qatlib-23.11.0-1.el9.x86_64.rpm 661 kB/s | 216 kB 00:00 2024-03-11T18:19:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:19:07.356 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:19:07.380 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:07.387 INFO:teuthology.orchestra.run.smithi165.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-11T18:19:07.398 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:19:07.412 INFO:teuthology.orchestra.run.smithi165.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 318 kB/s | 94 kB 00:00 2024-03-11T18:19:07.417 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:19:07.422 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:19:07.446 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:19:07.484 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:19:07.523 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:19:07.653 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:19:07.767 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:19:07.783 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:19:07.805 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:19:07.838 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:19:07.845 INFO:teuthology.orchestra.run.smithi165.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-11T18:19:07.862 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:07.863 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:19:07.928 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:19:07.954 INFO:teuthology.orchestra.run.smithi165.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.8 MB/s | 309 kB 00:00 2024-03-11T18:19:08.005 INFO:teuthology.orchestra.run.smithi165.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-11T18:19:08.016 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:08.047 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:08.120 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:19:08.123 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:08.157 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:19:08.176 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:19:08.190 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:08.256 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:08.263 INFO:teuthology.orchestra.run.smithi165.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-11T18:19:08.319 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:19:08.330 INFO:teuthology.orchestra.run.smithi165.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-03-11T18:19:08.357 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:19:08.380 INFO:teuthology.orchestra.run.smithi165.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-03-11T18:19:08.390 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:19:08.420 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:19:08.439 INFO:teuthology.orchestra.run.smithi165.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-11T18:19:08.448 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:19:08.483 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:19:08.514 INFO:teuthology.orchestra.run.smithi165.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-11T18:19:08.522 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:19:08.562 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:19:08.586 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:08.589 INFO:teuthology.orchestra.run.smithi165.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-11T18:19:08.591 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:19:08.632 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:08.671 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:19:08.673 INFO:teuthology.orchestra.run.smithi165.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-11T18:19:08.674 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:19:08.702 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:19:08.732 INFO:teuthology.orchestra.run.smithi165.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 747 kB/s | 43 kB 00:00 2024-03-11T18:19:08.735 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:19:08.777 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:19:08.790 INFO:teuthology.orchestra.run.smithi165.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-03-11T18:19:08.811 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:19:08.849 INFO:teuthology.orchestra.run.smithi165.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-11T18:19:08.905 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:19:08.907 INFO:teuthology.orchestra.run.smithi165.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-11T18:19:08.966 INFO:teuthology.orchestra.run.smithi165.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-03-11T18:19:08.994 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:19:09.022 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:19:09.050 INFO:teuthology.orchestra.run.smithi165.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-11T18:19:09.051 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:19:09.082 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:19:09.117 INFO:teuthology.orchestra.run.smithi165.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 529 kB/s | 35 kB 00:00 2024-03-11T18:19:09.130 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:19:09.180 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:09.184 INFO:teuthology.orchestra.run.smithi165.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-11T18:19:09.241 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:19:09.253 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:09.294 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:19:09.300 INFO:teuthology.orchestra.run.smithi165.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 18 MB/s | 2.0 MB 00:00 2024-03-11T18:19:09.384 INFO:teuthology.orchestra.run.smithi165.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-11T18:19:09.452 INFO:teuthology.orchestra.run.smithi165.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-11T18:19:09.489 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:09.527 INFO:teuthology.orchestra.run.smithi165.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-03-11T18:19:09.602 INFO:teuthology.orchestra.run.smithi165.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 309 kB/s | 23 kB 00:00 2024-03-11T18:19:09.608 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:09.653 INFO:teuthology.orchestra.run.smithi165.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-03-11T18:19:09.703 INFO:teuthology.orchestra.run.smithi165.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-03-11T18:19:09.736 INFO:teuthology.orchestra.run.smithi165.stdout:(130/174): protobuf-compiler-3.14.0-13.el9.x86_ 367 kB/s | 863 kB 00:02 2024-03-11T18:19:09.787 INFO:teuthology.orchestra.run.smithi165.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-11T18:19:09.837 INFO:teuthology.orchestra.run.smithi165.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-11T18:19:09.887 INFO:teuthology.orchestra.run.smithi165.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-11T18:19:09.913 INFO:teuthology.orchestra.run.smithi165.stdout:(134/174): python3-logutils-0.3.5-21.el9.noarch 263 kB/s | 46 kB 00:00 2024-03-11T18:19:09.950 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:09.969 INFO:teuthology.orchestra.run.smithi165.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.3 MB/s | 272 kB 00:00 2024-03-11T18:19:09.991 INFO:teuthology.orchestra.run.smithi165.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-11T18:19:10.030 INFO:teuthology.orchestra.run.smithi165.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.4 MB/s | 90 kB 00:00 2024-03-11T18:19:10.047 INFO:teuthology.orchestra.run.smithi165.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-11T18:19:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:19:10.081 INFO:teuthology.orchestra.run.smithi165.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-03-11T18:19:10.131 INFO:teuthology.orchestra.run.smithi165.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-11T18:19:10.147 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:19:10.181 INFO:teuthology.orchestra.run.smithi165.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-03-11T18:19:10.207 INFO:teuthology.orchestra.run.smithi165.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-03-11T18:19:10.240 INFO:teuthology.orchestra.run.smithi165.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-03-11T18:19:10.274 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:19:10.291 INFO:teuthology.orchestra.run.smithi165.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-11T18:19:10.314 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:19:10.316 INFO:teuthology.orchestra.run.smithi165.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-11T18:19:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:19:10.355 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:19:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:19:10.375 INFO:teuthology.orchestra.run.smithi165.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-03-11T18:19:10.402 INFO:teuthology.orchestra.run.smithi165.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-11T18:19:10.420 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:19:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:19:10.442 INFO:teuthology.orchestra.run.smithi165.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-11T18:19:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:19:10.534 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:19:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:19:10.559 INFO:teuthology.orchestra.run.smithi165.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-11T18:19:10.598 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:19:10.639 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:19:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:19:10.749 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:19:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:19:10.795 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:19:10.842 INFO:teuthology.orchestra.run.smithi165.stdout:(150/174): librbd1-19.0.0-1607.g94a53970.el9.x8 11 MB/s | 3.1 MB 00:00 2024-03-11T18:19:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:19:10.858 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:19:10.884 INFO:teuthology.orchestra.run.smithi165.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-11T18:19:10.908 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:19:10.948 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:19:11.027 INFO:teuthology.orchestra.run.smithi165.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-11T18:19:11.070 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:19:11.093 INFO:teuthology.orchestra.run.smithi165.stdout:(153/174): librados2-19.0.0-1607.g94a53970.el9. 5.2 MB/s | 3.4 MB 00:00 2024-03-11T18:19:11.110 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:19:11.152 INFO:teuthology.orchestra.run.smithi165.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 728 kB/s | 42 kB 00:00 2024-03-11T18:19:11.161 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:19:11.191 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:19:11.211 INFO:teuthology.orchestra.run.smithi165.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 9.9 MB/s | 1.8 MB 00:00 2024-03-11T18:19:11.253 INFO:teuthology.orchestra.run.smithi165.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-03-11T18:19:11.286 INFO:teuthology.orchestra.run.smithi165.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 947 kB/s | 71 kB 00:00 2024-03-11T18:19:11.302 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:19:11.328 INFO:teuthology.orchestra.run.smithi165.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 484 kB/s | 36 kB 00:00 2024-03-11T18:19:11.360 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:19:11.370 INFO:teuthology.orchestra.run.smithi165.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-11T18:19:11.413 INFO:teuthology.orchestra.run.smithi165.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 256 kB 00:00 2024-03-11T18:19:11.438 INFO:teuthology.orchestra.run.smithi165.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 346 kB/s | 23 kB 00:00 2024-03-11T18:19:11.463 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:19:11.496 INFO:teuthology.orchestra.run.smithi165.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-11T18:19:11.521 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:19:11.530 INFO:teuthology.orchestra.run.smithi165.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-11T18:19:11.572 INFO:teuthology.orchestra.run.smithi165.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-11T18:19:11.606 INFO:teuthology.orchestra.run.smithi165.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 850 kB/s | 64 kB 00:00 2024-03-11T18:19:11.633 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:19:11.639 INFO:teuthology.orchestra.run.smithi165.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 429 kB/s | 29 kB 00:00 2024-03-11T18:19:11.697 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:19:11.723 INFO:teuthology.orchestra.run.smithi165.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-03-11T18:19:11.764 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:19:11.808 INFO:teuthology.orchestra.run.smithi165.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 13 MB/s | 2.2 MB 00:00 2024-03-11T18:19:11.898 INFO:teuthology.orchestra.run.smithi165.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 13 MB/s | 2.3 MB 00:00 2024-03-11T18:19:11.932 INFO:teuthology.orchestra.run.smithi165.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.7 MB/s | 465 kB 00:00 2024-03-11T18:19:11.966 INFO:teuthology.orchestra.run.smithi165.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-03-11T18:19:12.057 INFO:teuthology.orchestra.run.smithi165.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-11T18:19:12.183 INFO:teuthology.orchestra.run.smithi165.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 343 kB/s | 74 kB 00:00 2024-03-11T18:19:12.480 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:19:12.582 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:19:12.687 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:19:12.745 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:12.771 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:12.823 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:19:12.898 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:19:12.933 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:19:12.952 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:19:13.006 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:19:13.028 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:19:13.036 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:19:13.067 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:19:13.095 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:19:13.421 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:19:13.440 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:19:13.458 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:13.550 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:13.588 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:19:13.641 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:19:13.652 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:19:13.735 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:19:13.792 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:19:14.007 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:19:14.065 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:19:14.124 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:19:14.137 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:19:14.153 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:14.224 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:14.238 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:19:14.254 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:19:14.281 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:19:14.320 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:19:14.389 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:19:14.429 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:19:14.492 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:19:14.559 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:19:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:19:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:19:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:19:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:19:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:19:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:19:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:19:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:19:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:19:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:19:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:19:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:19:15.716 INFO:teuthology.orchestra.run.smithi165.stdout:(174/174): openssl-devel-3.0.7-27.el9.x86_64.rp 452 kB/s | 4.1 MB 00:09 2024-03-11T18:19:15.721 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:19:15.721 INFO:teuthology.orchestra.run.smithi165.stdout:Total 12 MB/s | 223 MB 00:18 2024-03-11T18:19:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:19:15.829 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:19:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:19:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:19:16.205 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-03-11T18:19:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:19:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:19:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:19:16.490 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-03-11T18:19:16.490 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-03-11T18:19:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:19:16.526 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:19:16.568 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:19:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:19:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:19:16.750 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:19:16.797 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:19:16.929 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:19:16.976 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:19:17.027 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:19:17.098 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:19:17.136 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:17.139 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:19:17.171 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:17.252 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:17.253 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:19:17.253 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:19:17.253 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:17.280 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:17.287 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:19:17.409 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:19:17.468 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:19:17.503 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:19:17.563 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:19:17.736 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:19:17.803 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:19:18.333 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-03-11T18:19:18.334 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-03-11T18:19:18.778 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:19:18.836 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:19:18.869 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:19:18.926 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:19:19.042 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:19:19.077 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:19:19.316 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:19:19.357 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:19:19.448 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:19:20.299 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:19:20.434 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:19:22.045 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:19:22.136 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:19:22.174 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:19:22.230 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:19:22.271 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:19:22.671 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:19:22.702 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:22.823 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:22.859 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:19:22.916 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:19:23.462 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:19:23.755 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:23.863 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:23.891 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:19:25.381 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-03-11T18:19:25.511 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-11T18:19:25.545 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-11T18:19:25.585 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:25.595 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-11T18:19:25.634 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-11T18:19:25.687 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-11T18:19:25.721 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-11T18:19:25.747 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-11T18:19:25.781 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-11T18:19:26.106 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-11T18:19:26.173 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:26.294 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:26.324 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:26.416 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:19:26.487 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-11T18:19:26.521 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:26.561 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:26.660 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-11T18:19:26.709 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-11T18:19:26.738 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:26.824 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:26.847 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:26.894 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 17/198 2024-03-11T18:19:26.935 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-11T18:19:26.957 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-11T18:19:27.123 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:27.155 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-11T18:19:27.195 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-11T18:19:27.224 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:27.264 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:27.300 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-11T18:19:27.389 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:27.414 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:27.414 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:19:27.415 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:19:27.415 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:27.432 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-11T18:19:27.465 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:27.506 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:27.559 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:27.780 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-11T18:19:27.897 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-11T18:19:27.992 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-11T18:19:28.031 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-11T18:19:28.057 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-11T18:19:28.093 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-11T18:19:28.167 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-11T18:19:28.221 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-11T18:19:28.259 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-11T18:19:28.322 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-11T18:19:28.358 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-11T18:19:28.402 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-11T18:19:28.434 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-11T18:19:28.463 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-11T18:19:28.532 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-11T18:19:28.568 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 39/198 2024-03-11T18:19:28.611 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 40/198 2024-03-11T18:19:28.639 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 41/198 2024-03-11T18:19:28.708 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-11T18:19:28.763 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-11T18:19:28.836 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-11T18:19:28.970 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-11T18:19:29.031 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /root 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-11T18:19:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:29.501 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:29.519 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:29.519 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:29.520 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:19:29.520 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:29.552 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-11T18:19:29.636 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-11T18:19:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-11T18:19:29.764 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:29.772 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-11T18:19:29.804 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-11T18:19:29.845 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-11T18:19:29.883 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 53/198 2024-03-11T18:19:29.913 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-11T18:19:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-11T18:19:30.292 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-11T18:19:30.374 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-11T18:19:30.418 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-11T18:19:30.557 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-11T18:19:30.604 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-11T18:19:30.658 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-11T18:19:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-11T18:19:30.761 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-11T18:19:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-11T18:19:30.844 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-11T18:19:30.882 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-11T18:19:30.923 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-11T18:19:30.998 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-11T18:19:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:31.064 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:31.065 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:19:31.065 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:31.065 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:31.248 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-11T18:19:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:31.312 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:31.312 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:19:31.312 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:31.312 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:31.346 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-11T18:19:31.377 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-11T18:19:31.446 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-11T18:19:31.545 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-11T18:19:31.585 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-11T18:19:31.625 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-11T18:19:31.651 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-11T18:19:31.676 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-11T18:19:31.747 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-11T18:19:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:31.796 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:31.796 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:19:31.796 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:31.796 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:31.885 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-11T18:19:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:19:31.985 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-11T18:19:32.026 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-11T18:19:32.075 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-11T18:19:32.129 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-11T18:19:32.313 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-11T18:19:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:32.355 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-11T18:19:32.356 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:32.388 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:19:32.450 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:19:32.479 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:32.485 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-11T18:19:32.520 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-11T18:19:32.553 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-11T18:19:32.592 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-11T18:19:32.620 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-11T18:19:32.654 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-11T18:19:32.693 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-11T18:19:32.722 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:32.725 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-11T18:19:32.755 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:32.755 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-11T18:19:32.831 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-11T18:19:32.866 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-11T18:19:32.899 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-11T18:19:32.941 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-11T18:19:32.975 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-11T18:19:33.065 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-11T18:19:33.146 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-11T18:19:33.198 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-11T18:19:33.230 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-11T18:19:33.276 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-11T18:19:33.374 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-11T18:19:33.423 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-11T18:19:33.771 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:33.797 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:33.903 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:19:34.108 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:34.132 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:34.154 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-11T18:19:34.183 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:34.194 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:34.204 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-11T18:19:34.238 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-11T18:19:34.240 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:19:34.286 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-11T18:19:34.373 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-11T18:19:34.405 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-11T18:19:34.585 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-11T18:19:34.620 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-11T18:19:34.698 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-11T18:19:35.141 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:19:35.169 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:35.192 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:35.192 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:35.192 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:19:35.192 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:35.193 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:35.193 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:35.259 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:35.259 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:19:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:35.303 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:35.327 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:35.338 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-11T18:19:35.443 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-11T18:19:35.654 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:36.657 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-11T18:19:36.738 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-11T18:19:36.770 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-11T18:19:36.801 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-11T18:19:36.828 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-11T18:19:37.139 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-11T18:19:37.167 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:37.257 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 124/198 2024-03-11T18:19:37.288 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 125/198 2024-03-11T18:19:37.324 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-11T18:19:37.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-11T18:19:37.812 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:37.883 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 128/198 2024-03-11T18:19:37.905 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 129/198 2024-03-11T18:19:40.053 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:40.095 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:40.352 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:19:40.417 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:19:40.457 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:19:40.494 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:19:40.633 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 130/198 2024-03-11T18:19:40.673 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:40.693 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:19:40.754 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 131/198 2024-03-11T18:19:40.757 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T18:19:40.757 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T18:19:40.757 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:40.783 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:40.812 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:19:40.858 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:19:40.896 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:19:40.940 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:19:40.975 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:19:40.998 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:40.998 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2024-03-11T18:19:40.998 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2024-03-11T18:19:40.998 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2024-03-11T18:19:40.998 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2024-03-11T18:19:40.999 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2024-03-11T18:19:40.999 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /root 2024-03-11T18:19:40.999 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2024-03-11T18:19:40.999 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:41.008 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:19:41.057 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:19:41.095 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:19:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:19:41.235 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:19:41.252 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:19:41.305 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:41.306 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:41.306 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:19:41.306 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:41.881 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:41.924 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:41.945 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:42.842 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:19:42.928 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:19:42.969 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:19:42.999 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:43.052 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:43.082 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:43.089 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:19:43.114 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:19:43.140 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:19:43.166 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:19:43.167 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:19:43.254 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:19:43.292 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:19:43.292 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:19:43.331 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:43.386 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:19:43.412 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:19:43.417 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:43.417 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:43.417 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:19:43.418 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:43.418 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:43.418 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:43.445 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:19:43.470 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:19:43.495 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:19:43.528 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:19:43.576 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:19:43.616 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:19:43.642 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:19:43.667 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:19:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:19:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:19:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:43.921 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:43.921 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:19:43.932 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:44.005 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:44.005 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:44.005 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:19:44.005 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:44.006 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:44.006 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:44.012 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:19:44.054 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:19:44.087 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:19:44.120 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:19:44.126 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:19:44.146 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:19:44.213 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:19:44.761 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:44.793 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:44.844 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:19:44.951 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:19:45.014 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:45.365 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:46.586 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:46.607 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:46.758 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:46.945 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:19:46.976 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:47.051 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:47.109 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:47.148 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:19:48.284 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:19:48.320 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:48.400 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:48.471 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:48.556 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:48.556 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:19:48.556 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:48.638 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:48.726 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:49.122 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:49.132 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:19:49.132 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:19:49.132 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:19:49.132 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:19:49.132 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:19:49.133 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:19:49.134 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:19:49.135 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:19:49.135 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:19:49.151 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:19:49.152 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:19:49.153 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:19:49.154 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:19:49.155 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:19:49.156 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:19:49.157 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:19:49.158 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:19:49.159 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:19:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:19:49.161 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:19:49.162 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:19:49.163 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:19:49.164 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:19:49.164 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:19:49.164 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:19:49.212 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:49.212 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:49.213 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:19:49.213 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:49.213 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:49.213 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:19:52.182 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.183 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:19:52.184 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.185 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.186 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.187 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:19:52.188 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:19:52.189 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:19:52.190 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:19:52.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:19:52.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:19:52.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:19:52.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:19:52.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:19:52.197 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-11T18:19:52.198 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 132/198 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-03-11T18:19:52.359 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-03-11T18:19:52.360 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-03-11T18:19:52.360 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-03-11T18:19:52.360 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:52.525 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:52.540 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 133/198 2024-03-11T18:19:52.540 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:52.540 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T18:19:52.540 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:52.646 DEBUG:teuthology.parallel:result is None 2024-03-11T18:19:54.003 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 134/198 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T18:19:54.029 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:54.237 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:54.259 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 135/198 2024-03-11T18:19:54.260 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:54.260 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T18:19:54.260 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:54.260 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T18:19:54.260 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:54.288 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:19:54.346 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:19:54.396 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:19:54.434 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:19:54.705 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:54.727 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 136/198 2024-03-11T18:19:54.728 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:54.728 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T18:19:54.728 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:54.728 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T18:19:54.728 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:54.734 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:19:54.815 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-11T18:19:54.867 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:19:54.921 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:19:54.966 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:19:55.044 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:19:55.097 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:19:55.132 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:19:55.173 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:55.177 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:19:55.205 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 138/198 2024-03-11T18:19:55.229 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:19:55.236 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 139/198 2024-03-11T18:19:55.298 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 140/198 2024-03-11T18:19:55.313 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:19:55.328 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:55.446 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:19:55.486 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:19:55.558 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 141/198 2024-03-11T18:19:55.588 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:56.421 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:56.466 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:56.528 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:19:56.555 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 142/198 2024-03-11T18:19:56.580 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:56.679 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 143/198 2024-03-11T18:19:56.843 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 144/198 2024-03-11T18:19:56.882 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 145/198 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T18:19:56.905 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:56.949 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:56.960 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 146/198 2024-03-11T18:19:57.005 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 147/198 2024-03-11T18:19:57.865 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-11T18:19:57.901 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:57.924 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 149/198 2024-03-11T18:19:57.924 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:57.924 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T18:19:57.925 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:57.925 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T18:19:57.925 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:57.969 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53 150/198 2024-03-11T18:19:57.991 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:57.991 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T18:19:57.991 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:57.992 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:19:58.036 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:58.058 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:19:58.059 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:58.059 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-11T18:19:58.059 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:58.059 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-11T18:19:58.059 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:58.104 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:19:58.156 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:19:58.303 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:19:58.340 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:19:58.371 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:58.373 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:19:58.393 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 152/198 2024-03-11T18:19:58.393 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T18:19:58.393 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T18:19:58.394 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:58.394 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T18:19:58.394 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:19:58.408 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:19:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:19:58.452 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:19:58.596 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:19:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:19:58.654 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:19:58.791 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:19:58.836 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:19:58.898 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:19:58.940 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:19:58.972 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:19:59.047 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:19:59.123 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:19:59.171 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:19:59.204 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:19:59.237 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:19:59.271 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:19:59.304 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:19:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:59.342 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:59.679 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:19:59.680 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:19:59.818 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:19:59.849 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:19:59.881 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:19:59.942 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:19:59.975 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:20:00.132 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:20:00.396 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:20:00.396 INFO:teuthology.orchestra.run.smithi002.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-11T18:20:00.396 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:20:02.817 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:20:02.858 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 154/198 2024-03-11T18:20:02.888 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 155/198 2024-03-11T18:20:02.915 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-11T18:20:03.096 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 157/198 2024-03-11T18:20:03.228 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-11T18:20:03.272 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 159/198 2024-03-11T18:20:03.311 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-11T18:20:03.364 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-11T18:20:03.399 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-11T18:20:03.423 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 163/198 2024-03-11T18:20:03.456 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-11T18:20:03.485 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-11T18:20:03.540 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-11T18:20:03.600 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-11T18:20:03.634 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-11T18:20:04.232 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:20:04.266 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:20:04.286 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-11T18:20:05.107 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 170/198 2024-03-11T18:20:05.187 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-11T18:20:05.227 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-11T18:20:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:20:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:20:05.257 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:20:05.258 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:20:05.259 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:20:05.277 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:20:05.278 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:20:05.279 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:20:05.280 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:20:05.281 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:20:05.282 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:20:05.283 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:20:05.284 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:20:05.285 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:20:05.286 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:20:05.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:20:05.288 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:20:05.289 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:20:05.290 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:20:05.308 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-11T18:20:05.349 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-11T18:20:05.374 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-11T18:20:05.399 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-11T18:20:05.426 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-11T18:20:05.426 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:20:05.514 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-11T18:20:05.544 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-11T18:20:05.544 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:20:05.631 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-11T18:20:05.655 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-11T18:20:05.688 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-11T18:20:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-11T18:20:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-11T18:20:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-11T18:20:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-11T18:20:05.869 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-11T18:20:05.894 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-11T18:20:05.919 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-11T18:20:05.944 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:20:05.969 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-11T18:20:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:20:05.976 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:20:06.156 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-11T18:20:06.156 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:20:06.222 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-11T18:20:06.247 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:20:06.272 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-11T18:20:06.305 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-11T18:20:06.332 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-11T18:20:06.395 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:07.938 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.939 INFO:teuthology.orchestra.run.smithi002.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:20:07.940 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.941 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:07.942 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.943 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:20:07.944 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:20:07.945 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:20:07.946 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:20:07.947 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:20:07.948 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:20:07.949 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:20:07.950 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:20:07.951 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:20:07.952 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-03-11T18:20:08.550 DEBUG:teuthology.parallel:result is None 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 3/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 4/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53 5/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 6/198 2024-03-11T18:20:11.317 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 7/198 2024-03-11T18:20:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 8/198 2024-03-11T18:20:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 9/198 2024-03-11T18:20:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 10/198 2024-03-11T18:20:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 11/198 2024-03-11T18:20:11.318 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 12/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 13/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 14/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_ 15/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 16/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 17/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_6 18/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86 19/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 20/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el 21/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9. 22/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_6 23/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 24/198 2024-03-11T18:20:11.319 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 25/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 26/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 27/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 28/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 29/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970. 30/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noa 31/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.n 32/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a 33/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el 34/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 35/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.e 36/198 2024-03-11T18:20:11.320 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 37/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 38/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 39/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-11T18:20:11.321 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-11T18:20:11.322 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-11T18:20:11.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-11T18:20:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-11T18:20:11.325 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-11T18:20:11.326 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-11T18:20:11.327 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-11T18:20:11.328 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 151/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 153/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-11T18:20:11.329 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-11T18:20:11.330 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-11T18:20:11.331 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-11T18:20:11.332 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-11T18:20:14.064 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-11T18:20:14.064 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:20:14.064 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-03-11T18:20:14.064 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: acl-2.3.1-4.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.065 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: automake-1.16.2-8.el9.noarch 2024-03-11T18:20:14.066 INFO:teuthology.orchestra.run.smithi165.stdout: bison-3.7.4-5.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.067 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-shell-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.068 INFO:teuthology.orchestra.run.smithi165.stdout: cephfs-top-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: flex-2.6.4-9.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: jq-1.6-16.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T18:20:14.069 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.070 INFO:teuthology.orchestra.run.smithi165.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-13.11-1.el9.x86_64 2024-03-11T18:20:14.071 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T18:20:14.072 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T18:20:14.073 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T18:20:14.074 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T18:20:14.075 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T18:20:14.076 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T18:20:14.077 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-11T18:20:14.078 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T18:20:14.079 INFO:teuthology.orchestra.run.smithi165.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-11T18:20:14.079 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:20:14.079 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-03-11T18:20:14.536 DEBUG:teuthology.parallel:result is None 2024-03-11T18:20:14.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:14.687 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:20:14.753 INFO:teuthology.orchestra.run.smithi002.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:20:14.753 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:20:14.754 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:20:14.755 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:15.020 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:20:15.047 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:20:15.047 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:20:15.047 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:20:15.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:15.194 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T18:20:15.215 INFO:teuthology.orchestra.run.smithi165.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T18:20:15.215 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T18:20:15.216 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T18:20:15.217 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T18:20:15.218 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:20:15.218 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:20:15.309 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:20:15.309 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:20:15.351 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:20:15.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T18:20:15.381 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T18:20:15.381 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:20:15.381 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:20:15.467 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:20:15.594 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:20:15.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:20:15.632 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:20:15.705 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:20:15.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T18:20:15.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T18:20:15.798 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T18:20:15.798 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:20:15.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:20:15.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:20:16.007 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:20:16.007 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:20:16.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:20:16.110 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:20:16.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T18:20:16.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T18:20:16.206 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T18:20:16.206 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:20:16.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:20:16.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:20:16.376 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:20:16.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:20:16.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:20:16.477 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:20:16.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T18:20:16.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T18:20:16.573 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T18:20:16.674 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'subvols': {'create': 2}} 2024-03-11T18:20:16.674 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:16.675 INFO:tasks.cephadm:Cluster fsid is 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:20:16.675 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T18:20:16.675 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.b': '172.21.15.23', 'mon.c': '172.21.15.165'} 2024-03-11T18:20:16.675 INFO:tasks.cephadm:First mon is mon.a on smithi002 2024-03-11T18:20:16.675 INFO:tasks.cephadm:First mgr is x 2024-03-11T18:20:16.675 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T18:20:16.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2024-03-11T18:20:16.769 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-11T18:20:16.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-03-11T18:20:16.828 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T18:20:16.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:16.968 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T18:20:17.110 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:20:17.111 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T18:20:17.111 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:17.749 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:20 /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:17.750 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.153 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:20 /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.153 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.708 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 18:20 /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.709 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.775 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.794 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T18:20:18.830 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T18:20:18.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:20:18.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:20:18.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T18:20:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:20:19.818 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:20:20.097 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:20:46.231 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T18:20:46.231 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:20:46.231 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:20:46.232 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-11T18:20:46.232 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:20:46.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-11T18:20:46.232 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T18:20:50.779 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-03-11T18:20:50.780 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T18:20:57.196 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:20:57.196 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:20:57.196 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T18:20:57.196 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2024-03-11T18:20:57.196 INFO:teuthology.orchestra.run.smithi002.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T18:20:57.197 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:20:57.197 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:20:57.243 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2024-03-11T18:20:57.349 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-11T18:20:57.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-03-11T18:20:57.425 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2024-03-11T18:20:57.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-11T18:20:57.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-03-11T18:20:57.598 INFO:tasks.cephadm:Writing seed config... 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-11T18:20:57.599 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-11T18:20:57.600 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T18:20:57.601 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T18:20:57.602 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-11T18:20:57.603 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:20:57.603 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T18:20:57.663 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 02b83714-dfd4-11ee-95c8-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-11T18:20:57.663 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T18:20:57.711 DEBUG:teuthology.orchestra.run.smithi002:mgr.x> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T18:20:57.757 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T18:20:57.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:20:58.034 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-03-11T18:20:58.035 INFO:teuthology.orchestra.run.smithi002.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', '02b83714-dfd4-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2024-03-11T18:20:58.094 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:20:58.095 INFO:teuthology.orchestra.run.smithi002.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T18:20:58.095 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying podman|docker is present... 2024-03-11T18:20:58.150 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:20:58.151 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying lvm2 is present... 2024-03-11T18:20:58.151 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying time synchronization is in place... 2024-03-11T18:20:58.166 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T18:20:58.166 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T18:20:58.185 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T18:20:58.185 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:20:58.210 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-03-11T18:20:58.232 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-03-11T18:20:58.232 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-03-11T18:20:58.233 INFO:teuthology.orchestra.run.smithi002.stdout:Repeating the final host check... 2024-03-11T18:20:58.290 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T18:20:58.290 INFO:teuthology.orchestra.run.smithi002.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T18:20:58.290 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl is present 2024-03-11T18:20:58.290 INFO:teuthology.orchestra.run.smithi002.stdout:lvcreate is present 2024-03-11T18:20:58.304 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T18:20:58.304 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T18:20:58.320 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T18:20:58.320 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:20:58.343 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout enabled 2024-03-11T18:20:58.363 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout active 2024-03-11T18:20:58.363 INFO:teuthology.orchestra.run.smithi002.stdout:Unit chronyd.service is enabled and running 2024-03-11T18:20:58.364 INFO:teuthology.orchestra.run.smithi002.stdout:Host looks OK 2024-03-11T18:20:58.364 INFO:teuthology.orchestra.run.smithi002.stdout:Cluster fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:20:58.364 INFO:teuthology.orchestra.run.smithi002.stdout:Acquiring lock 140338547580880 on /run/cephadm/02b83714-dfd4-11ee-95c8-87774f69a715.lock 2024-03-11T18:20:58.364 INFO:teuthology.orchestra.run.smithi002.stdout:Lock 140338547580880 acquired on /run/cephadm/02b83714-dfd4-11ee-95c8-87774f69a715.lock 2024-03-11T18:20:58.366 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 3300 ... 2024-03-11T18:20:58.367 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying IP 172.21.15.2 port 6789 ... 2024-03-11T18:20:58.367 INFO:teuthology.orchestra.run.smithi002.stdout:Base mon IP(s) is [172.21.15.2:3300, 172.21.15.2:6789], mon addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-03-11T18:20:58.381 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.2 metric 100 2024-03-11T18:20:58.382 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2024-03-11T18:20:58.394 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T18:20:58.394 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T18:20:58.394 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-03-11T18:20:58.405 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T18:20:58.409 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-03-11T18:20:58.409 INFO:teuthology.orchestra.run.smithi002.stdout:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-03-11T18:20:58.409 INFO:teuthology.orchestra.run.smithi002.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T18:20:58.411 INFO:teuthology.orchestra.run.smithi002.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T18:20:58.414 INFO:teuthology.orchestra.run.smithi002.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T18:20:58.997 INFO:teuthology.orchestra.run.smithi002.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T18:21:11.048 INFO:teuthology.orchestra.run.smithi002.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T18:21:11.048 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T18:21:11.048 INFO:teuthology.orchestra.run.smithi002.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T18:21:12.143 INFO:teuthology.orchestra.run.smithi002.stdout:stat: stdout 167 167 2024-03-11T18:21:12.144 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial keys... 2024-03-11T18:21:13.146 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCYS+9lWg7RIBAAVMT5XLguAYGX2o8WVCy92A== 2024-03-11T18:21:14.244 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCZS+9lrn3eJBAA13gO4uq4X6wwxcnnqGXkeg== 2024-03-11T18:21:15.326 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph-authtool: stdout AQCaS+9l5OxIKRAAvdHR43a3+rFpK1HBnvX4dQ== 2024-03-11T18:21:15.329 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial monmap... 2024-03-11T18:21:16.830 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = quincy 2024-03-11T18:21:16.831 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: set fsid to 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:16.832 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T18:21:16.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:21:16.832 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mon... 2024-03-11T18:21:17.889 INFO:teuthology.orchestra.run.smithi002.stdout:create mon.a on 2024-03-11T18:21:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T18:21:18.504 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T18:21:18.793 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target. 2024-03-11T18:21:18.793 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target. 2024-03-11T18:21:19.229 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a 2024-03-11T18:21:19.230 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T18:21:19.526 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target.wants/ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715@.service. 2024-03-11T18:21:20.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:19 smithi002 podman[30948]: 2024-03-11 18:21:19.926935774 +0000 UTC m=+0.131743556 container create 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T18:21:20.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 podman[30948]: 2024-03-11 18:21:20.19001827 +0000 UTC m=+0.394826097 container init 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T18:21:20.307 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:21:20.307 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T18:21:20.325 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:21:20.325 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:21:20.325 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T18:21:20.325 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T18:21:20.326 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon to start... 2024-03-11T18:21:20.326 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mon... 2024-03-11T18:21:20.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 podman[30948]: 2024-03-11 18:21:20.204865144 +0000 UTC m=+0.409672937 container start 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:21:20.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 bash[30948]: 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 2024-03-11T18:21:20.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 systemd[1]: Started Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:21:20.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 ceph-mon[30966]: mkfs 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:20.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:20 smithi002 ceph-mon[30966]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T18:21:21.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T18:21:21.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:21:21.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: fsmap 2024-03-11T18:21:21.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T18:21:21.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: mgrmap e1: no daemons active 2024-03-11T18:21:21.627 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:21 smithi002 ceph-mon[30966]: from='client.? 172.21.15.2:0/803508116' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T18:21:21.821 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T18:21:21.821 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout id: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout services: 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.694265s) 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout data: 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T18:21:21.822 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.885+0000 7f1727600700 1 Processor -- start 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.885+0000 7f1727600700 1 -- start start 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.885+0000 7f1727600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.885+0000 7f1727600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172007ae10 con 0x7f1720105e60 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.886+0000 7f1724c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.886+0000 7f1724c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44610/0 (socket says 172.21.15.2:44610) 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.886+0000 7f1724c00700 1 -- 172.21.15.2:0/469318992 learned_addr learned my addr 172.21.15.2:0/469318992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.887+0000 7f1724c00700 1 -- 172.21.15.2:0/469318992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17201067c0 con 0x7f1720105e60 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.887+0000 7f1724c00700 1 --2- 172.21.15.2:0/469318992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17140098a0 tx=0x7f1714009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f126d21692c11b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.888+0000 7f171f400700 1 -- 172.21.15.2:0/469318992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1714004160 con 0x7f1720105e60 2024-03-11T18:21:21.823 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.888+0000 7f171f400700 1 -- 172.21.15.2:0/469318992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17140042c0 con 0x7f1720105e60 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.888+0000 7f171f400700 1 -- 172.21.15.2:0/469318992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17140045a0 con 0x7f1720105e60 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.888+0000 7f1727600700 1 -- 172.21.15.2:0/469318992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 msgr2=0x7f1720106280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.888+0000 7f1727600700 1 --2- 172.21.15.2:0/469318992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17140098a0 tx=0x7f1714009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.889+0000 7f1727600700 1 -- 172.21.15.2:0/469318992 shutdown_connections 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.889+0000 7f1727600700 1 --2- 172.21.15.2:0/469318992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f1720106280 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.889+0000 7f1727600700 1 -- 172.21.15.2:0/469318992 >> 172.21.15.2:0/469318992 conn(0x7f1720101aa0 msgr2=0x7f1720103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.889+0000 7f1727600700 1 -- 172.21.15.2:0/469318992 shutdown_connections 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.889+0000 7f1727600700 1 -- 172.21.15.2:0/469318992 wait complete. 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.890+0000 7f1727600700 1 Processor -- start 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.890+0000 7f1727600700 1 -- start start 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.891+0000 7f1727600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:21.824 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.891+0000 7f1727600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f172019d1d0 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.891+0000 7f1724c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.891+0000 7f1724c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44616/0 (socket says 172.21.15.2:44616) 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.891+0000 7f1724c00700 1 -- 172.21.15.2:0/803508116 learned_addr learned my addr 172.21.15.2:0/803508116 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f1724c00700 1 -- 172.21.15.2:0/803508116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009550 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f1724c00700 1 --2- 172.21.15.2:0/803508116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f172007a250 tx=0x7f1714004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1714011630 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1714011790 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1714011a30 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172019d3d0 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.892+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172019d870 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.893+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1714021070 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.893+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f171401b7d0 con 0x7f1720105e60 2024-03-11T18:21:21.825 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.894+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1720106b30 con 0x7f1720105e60 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.897+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f171401b990 con 0x7f1720105e60 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.979+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1720196360 con 0x7f1720105e60 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.980+0000 7f171d600700 1 -- 172.21.15.2:0/803508116 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f171401bb30 con 0x7f1720105e60 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.981+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 msgr2=0x7f172019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.981+0000 7f1727600700 1 --2- 172.21.15.2:0/803508116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f172007a250 tx=0x7f1714004750 comp rx=0 tx=0).stop 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.981+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 shutdown_connections 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.981+0000 7f1727600700 1 --2- 172.21.15.2:0/803508116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1720105e60 0x7f172019cc90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.981+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 >> 172.21.15.2:0/803508116 conn(0x7f1720101aa0 msgr2=0x7f1720193420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.982+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 shutdown_connections 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:20.982+0000 7f1727600700 1 -- 172.21.15.2:0/803508116 wait complete. 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:mon is available 2024-03-11T18:21:21.826 INFO:teuthology.orchestra.run.smithi002.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T18:21:22.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:22 smithi002 ceph-mon[30966]: from='client.? 172.21.15.2:0/1411496366' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T18:21:22.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:22 smithi002 ceph-mon[30966]: from='client.? 172.21.15.2:0/1411496366' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T18:21:23.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:23.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T18:21:23.038 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T18:21:23.040 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T18:21:23.041 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.372+0000 7f671fe00700 1 Processor -- start 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.372+0000 7f671fe00700 1 -- start start 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.373+0000 7f671fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.373+0000 7f671fe00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672007adb0 con 0x7f6720107e80 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.373+0000 7f671ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.373+0000 7f671ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44624/0 (socket says 172.21.15.2:44624) 2024-03-11T18:21:23.044 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.373+0000 7f671ea00700 1 -- 172.21.15.2:0/2203977803 learned_addr learned my addr 172.21.15.2:0/2203977803 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.374+0000 7f671ea00700 1 -- 172.21.15.2:0/2203977803 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672007aef0 con 0x7f6720107e80 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.374+0000 7f671ea00700 1 --2- 172.21.15.2:0/2203977803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24e5a6502ea00a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.374+0000 7f671d600700 1 -- 172.21.15.2:0/2203977803 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6710004160 con 0x7f6720107e80 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.374+0000 7f671d600700 1 -- 172.21.15.2:0/2203977803 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67100042c0 con 0x7f6720107e80 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.374+0000 7f671d600700 1 -- 172.21.15.2:0/2203977803 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67100045a0 con 0x7f6720107e80 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 -- 172.21.15.2:0/2203977803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 msgr2=0x7f672010a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 --2- 172.21.15.2:0/2203977803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 -- 172.21.15.2:0/2203977803 shutdown_connections 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 --2- 172.21.15.2:0/2203977803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672010a270 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 -- 172.21.15.2:0/2203977803 >> 172.21.15.2:0/2203977803 conn(0x7f67201017f0 msgr2=0x7f6720103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 -- 172.21.15.2:0/2203977803 shutdown_connections 2024-03-11T18:21:23.045 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.375+0000 7f671fe00700 1 -- 172.21.15.2:0/2203977803 wait complete. 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.376+0000 7f671fe00700 1 Processor -- start 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.376+0000 7f671fe00700 1 -- start start 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671fe00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672019ad80 con 0x7f6720107e80 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44632/0 (socket says 172.21.15.2:44632) 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671ea00700 1 -- 172.21.15.2:0/1411496366 learned_addr learned my addr 172.21.15.2:0/1411496366 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:23.046 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.377+0000 7f671ea00700 1 -- 172.21.15.2:0/1411496366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6710009550 con 0x7f6720107e80 2024-03-11T18:21:23.048 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f671ea00700 1 --2- 172.21.15.2:0/1411496366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f672007a1f0 tx=0x7f6710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:23.048 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6710011630 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710011790 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6710011a30 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672019af80 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672019b420 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.378+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6710022070 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.379+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f671001c450 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.379+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6720194630 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.382+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6710011b90 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.461+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f672010a990 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.468+0000 7f6724a00700 1 -- 172.21.15.2:0/1411496366 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f6710011de0 con 0x7f6720107e80 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 msgr2=0x7f672019a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 --2- 172.21.15.2:0/1411496366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f672007a1f0 tx=0x7f6710004750 comp rx=0 tx=0).stop 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 shutdown_connections 2024-03-11T18:21:23.049 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 --2- 172.21.15.2:0/1411496366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6720107e80 0x7f672019a840 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 >> 172.21.15.2:0/1411496366 conn(0x7f67201017f0 msgr2=0x7f6720103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 shutdown_connections 2024-03-11T18:21:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:22.470+0000 7f671fe00700 1 -- 172.21.15.2:0/1411496366 wait complete. 2024-03-11T18:21:23.050 INFO:teuthology.orchestra.run.smithi002.stdout:Generating new minimal ceph.conf... 2024-03-11T18:21:24.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:23 smithi002 ceph-mon[30966]: from='client.? 172.21.15.2:0/1607631844' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:21:24.368 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.611+0000 7f9913800700 1 Processor -- start 2024-03-11T18:21:24.368 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.611+0000 7f9913800700 1 -- start start 2024-03-11T18:21:24.368 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.612+0000 7f9913800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:24.368 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.612+0000 7f9913800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c07add0 con 0x7f990c1080b0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.612+0000 7f9910e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.613+0000 7f9910e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44644/0 (socket says 172.21.15.2:44644) 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.613+0000 7f9910e00700 1 -- 172.21.15.2:0/237333772 learned_addr learned my addr 172.21.15.2:0/237333772 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.613+0000 7f9910e00700 1 -- 172.21.15.2:0/237333772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c07af10 con 0x7f990c1080b0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.613+0000 7f9910e00700 1 --2- 172.21.15.2:0/237333772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f99000098a0 tx=0x7f9900009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53ed8fc6d36a52e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.614+0000 7f990b400700 1 -- 172.21.15.2:0/237333772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9900004160 con 0x7f990c1080b0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.614+0000 7f990b400700 1 -- 172.21.15.2:0/237333772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99000042c0 con 0x7f990c1080b0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.614+0000 7f990b400700 1 -- 172.21.15.2:0/237333772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9900011ac0 con 0x7f990c1080b0 2024-03-11T18:21:24.369 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 -- 172.21.15.2:0/237333772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 msgr2=0x7f990c10a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 --2- 172.21.15.2:0/237333772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f99000098a0 tx=0x7f9900009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 -- 172.21.15.2:0/237333772 shutdown_connections 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 --2- 172.21.15.2:0/237333772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c10a4a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 -- 172.21.15.2:0/237333772 >> 172.21.15.2:0/237333772 conn(0x7f990c101aa0 msgr2=0x7f990c103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 -- 172.21.15.2:0/237333772 shutdown_connections 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.615+0000 7f9913800700 1 -- 172.21.15.2:0/237333772 wait complete. 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.616+0000 7f9913800700 1 Processor -- start 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9913800700 1 -- start start 2024-03-11T18:21:24.373 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9913800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9913800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c19b8a0 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9910e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9910e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44660/0 (socket says 172.21.15.2:44660) 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.617+0000 7f9910e00700 1 -- 172.21.15.2:0/1607631844 learned_addr learned my addr 172.21.15.2:0/1607631844 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9910e00700 1 -- 172.21.15.2:0/1607631844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900009550 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9910e00700 1 --2- 172.21.15.2:0/1607631844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9900009fb0 tx=0x7f9900011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99000038b0 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9900003a10 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99000195e0 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.618+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c19baa0 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.619+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c19bf40 con 0x7f990c1080b0 2024-03-11T18:21:24.374 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.619+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9900019be0 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.619+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9900024730 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.620+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c10b770 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.624+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9900003b80 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.702+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f990c19eaa0 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.702+0000 7f9909600700 1 -- 172.21.15.2:0/1607631844 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f990002dd90 con 0x7f990c1080b0 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.704+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 msgr2=0x7f990c19b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.704+0000 7f9913800700 1 --2- 172.21.15.2:0/1607631844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9900009fb0 tx=0x7f9900011b20 comp rx=0 tx=0).stop 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.704+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 shutdown_connections 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.704+0000 7f9913800700 1 --2- 172.21.15.2:0/1607631844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f990c1080b0 0x7f990c19b360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.705+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 >> 172.21.15.2:0/1607631844 conn(0x7f990c101aa0 msgr2=0x7f990c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.705+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 shutdown_connections 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:23.705+0000 7f9913800700 1 -- 172.21.15.2:0/1607631844 wait complete. 2024-03-11T18:21:24.375 INFO:teuthology.orchestra.run.smithi002.stdout:Restarting the monitor... 2024-03-11T18:21:24.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:24 smithi002 systemd[1]: Stopping Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:21:25.199 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a[30962]: 2024-03-11T18:21:24.882+0000 7fbf2c600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:21:25.199 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a[30962]: 2024-03-11T18:21:24.882+0000 7fbf2c600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T18:21:25.199 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:24 smithi002 podman[31208]: 2024-03-11 18:21:24.924256798 +0000 UTC m=+0.151907127 container died 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T18:21:25.468 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 podman[31208]: 2024-03-11 18:21:25.30834797 +0000 UTC m=+0.535998328 container cleanup 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T18:21:25.468 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 bash[31208]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a 2024-03-11T18:21:25.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 podman[31221]: 2024-03-11 18:21:25.467337476 +0000 UTC m=+0.534323815 container remove 9d8334453c3d77960dd14e4f49bc14c5c288a5d27f6653cb0f549fa7b2bf8c65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T18:21:26.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T18:21:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 systemd[1]: Stopped Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:21:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service: Consumed 1.958s CPU time. 2024-03-11T18:21:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:25 smithi002 systemd[1]: Starting Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:21:26.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 podman[31310]: 2024-03-11 18:21:26.215251871 +0000 UTC m=+0.033596066 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:21:26.673 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 podman[31310]: 2024-03-11 18:21:26.672066978 +0000 UTC m=+0.490411228 container create e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T18:21:26.925 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 podman[31310]: 2024-03-11 18:21:26.913939699 +0000 UTC m=+0.732283936 container init e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T18:21:27.045 INFO:teuthology.orchestra.run.smithi002.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T18:21:27.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 podman[31310]: 2024-03-11 18:21:26.925072136 +0000 UTC m=+0.743416355 container start e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T18:21:27.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T18:21:27.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T18:21:27.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: pidfile_write: ignore empty --pid-file 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: load: jerasure load: lrc 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: RocksDB version: 7.9.2 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Git sha 0 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: DB SUMMARY 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: DB Session ID: E6100ZFCNBWF1UW1MCVN 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: CURRENT file: CURRENT 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99712 ; 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.error_if_exists: 0 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.create_if_missing: 0 2024-03-11T18:21:27.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.paranoid_checks: 1 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.env: 0x55ebce288ae0 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.info_log: 0x55ebd0129900 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.statistics: (nil) 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.use_fsync: 0 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_log_file_size: 0 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T18:21:27.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_fallocate: 1 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.use_direct_reads: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T18:21:27.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.db_log_dir: 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.wal_dir: 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.write_buffer_manager: 0x55ebd00fd400 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T18:21:27.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.unordered_write: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.row_cache: None 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.wal_filter: None 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T18:21:27.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.two_write_queues: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.wal_compression: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.atomic_flush: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T18:21:27.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.log_readahead_size: 0 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_background_jobs: 2 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_background_compactions: -1 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_subcompactions: 1 2024-03-11T18:21:27.215 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_open_files: -1 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_background_flushes: -1 2024-03-11T18:21:27.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Compression algorithms supported: 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kZSTD supported: 0 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kXpressCompression supported: 0 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kZlibCompression supported: 1 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kSnappyCompression supported: 1 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kLZ4Compression supported: 1 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: kBZip2Compression supported: 0 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T18:21:27.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.merge_operator: 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_filter: None 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ebd0129a00) 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-03-11T18:21:27.218 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55ebd012d090 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-03-11T18:21:27.219 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-03-11T18:21:27.220 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression: NoCompression 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.num_levels: 7 2024-03-11T18:21:27.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T18:21:27.222 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T18:21:27.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T18:21:27.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T18:21:27.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.table_properties_collectors: 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.inplace_update_support: 0 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T18:21:27.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.bloom_locality: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.max_successive_merges: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.ttl: 2592000 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enable_blob_files: false 2024-03-11T18:21:27.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.min_blob_size: 0 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2e7a4008-1391-4f7e-8485-bd60b395f8bd 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181286983950, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T18:21:27.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181286985686, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94893, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 92891, "index_size": 284, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13781, "raw_average_key_size": 50, "raw_value_size": 85392, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710181286, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2e7a4008-1391-4f7e-8485-bd60b395f8bd", "db_session_id": "E6100ZFCNBWF1UW1MCVN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181286985839, "job": 1, "event": "recovery_finished"} 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ebd021e000 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: DB pointer 0x55ebd0208000 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:21:27.229 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 94.50 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 94.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 82.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-03-11T18:21:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 82.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:21:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 13.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x55ebd012d090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???) e1 preinit fsid 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).mds e1 new map 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).mds e1 print_map 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-03-11T18:21:27.232 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:26 smithi002 ceph-mon[31328]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 bash[31310]: e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 systemd[1]: Started Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 ceph-mon[31328]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 ceph-mon[31328]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 ceph-mon[31328]: fsmap 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 ceph-mon[31328]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T18:21:27.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:27 smithi002 ceph-mon[31328]: mgrmap e1: no daemons active 2024-03-11T18:21:28.386 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.649+0000 7f7113c00700 1 Processor -- start 2024-03-11T18:21:28.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.650+0000 7f7113c00700 1 -- start start 2024-03-11T18:21:28.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.650+0000 7f7113c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:28.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.650+0000 7f7113c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c07ae90 con 0x7f710c107fe0 2024-03-11T18:21:28.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.651+0000 7f7111200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:28.389 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.651+0000 7f7111200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56270/0 (socket says 172.21.15.2:56270) 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.651+0000 7f7111200700 1 -- 172.21.15.2:0/1697745182 learned_addr learned my addr 172.21.15.2:0/1697745182 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.652+0000 7f7111200700 1 -- 172.21.15.2:0/1697745182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c10a910 con 0x7f710c107fe0 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.652+0000 7f7111200700 1 --2- 172.21.15.2:0/1697745182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f71000098a0 tx=0x7f7100009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d3049ec2602b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.652+0000 7f710b400700 1 -- 172.21.15.2:0/1697745182 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7100004160 con 0x7f710c107fe0 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.652+0000 7f710b400700 1 -- 172.21.15.2:0/1697745182 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71000042c0 con 0x7f710c107fe0 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.653+0000 7f710b400700 1 -- 172.21.15.2:0/1697745182 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7100011ac0 con 0x7f710c107fe0 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 -- 172.21.15.2:0/1697745182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 msgr2=0x7f710c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 --2- 172.21.15.2:0/1697745182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f71000098a0 tx=0x7f7100009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 -- 172.21.15.2:0/1697745182 shutdown_connections 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 --2- 172.21.15.2:0/1697745182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c10a3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 -- 172.21.15.2:0/1697745182 >> 172.21.15.2:0/1697745182 conn(0x7f710c101a50 msgr2=0x7f710c103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 -- 172.21.15.2:0/1697745182 shutdown_connections 2024-03-11T18:21:28.390 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.654+0000 7f7113c00700 1 -- 172.21.15.2:0/1697745182 wait complete. 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.655+0000 7f7113c00700 1 Processor -- start 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.655+0000 7f7113c00700 1 -- start start 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.656+0000 7f7113c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.656+0000 7f7113c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c19da30 con 0x7f710c107fe0 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.656+0000 7f7111200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.656+0000 7f7111200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56272/0 (socket says 172.21.15.2:56272) 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.656+0000 7f7111200700 1 -- 172.21.15.2:0/4114783740 learned_addr learned my addr 172.21.15.2:0/4114783740 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:28.391 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.657+0000 7f7111200700 1 -- 172.21.15.2:0/4114783740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100009550 con 0x7f710c107fe0 2024-03-11T18:21:28.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.657+0000 7f7111200700 1 --2- 172.21.15.2:0/4114783740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7100011a70 tx=0x7f7100011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:28.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.657+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71000039c0 con 0x7f710c107fe0 2024-03-11T18:21:28.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.657+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c19dc30 con 0x7f710c107fe0 2024-03-11T18:21:28.393 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.658+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7100003b20 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.658+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c19e0d0 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.658+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7100024620 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.658+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7100024800 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.659+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7100019700 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.659+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c10b770 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.665+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f710001a7c0 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.744+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f710c10b980 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.747+0000 7f7109600700 1 -- 172.21.15.2:0/4114783740 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f710002dd00 con 0x7f710c107fe0 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.748+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 msgr2=0x7f710c19d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.748+0000 7f7113c00700 1 --2- 172.21.15.2:0/4114783740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7100011a70 tx=0x7f7100011b50 comp rx=0 tx=0).stop 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.748+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 shutdown_connections 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.748+0000 7f7113c00700 1 --2- 172.21.15.2:0/4114783740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f710c107fe0 0x7f710c19d4f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:28.394 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.748+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 >> 172.21.15.2:0/4114783740 conn(0x7f710c101a50 msgr2=0x7f710c1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:28.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.749+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 shutdown_connections 2024-03-11T18:21:28.395 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:27.749+0000 7f7113c00700 1 -- 172.21.15.2:0/4114783740 wait complete. 2024-03-11T18:21:28.395 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T18:21:28.409 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:21:28.410 INFO:teuthology.orchestra.run.smithi002.stdout:Creating mgr... 2024-03-11T18:21:28.411 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T18:21:28.412 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T18:21:28.795 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x 2024-03-11T18:21:28.795 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Failed to reset failed state of unit ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x.service: Unit ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x.service not loaded. 2024-03-11T18:21:29.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:28 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4114783740' entity='client.admin' 2024-03-11T18:21:29.055 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715.target.wants/ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-02b83714-dfd4-11ee-95c8-87774f69a715@.service. 2024-03-11T18:21:29.432 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 systemd[1]: Starting Ceph mgr.x for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:21:29.706 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 podman[31552]: 2024-03-11 18:21:29.33928295 +0000 UTC m=+0.039052610 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T18:21:29.706 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 podman[31552]: 2024-03-11 18:21:29.43715572 +0000 UTC m=+0.136925415 container create b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T18:21:29.706 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 podman[31552]: 2024-03-11 18:21:29.63219021 +0000 UTC m=+0.331959909 container init b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T18:21:29.707 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 podman[31552]: 2024-03-11 18:21:29.641954179 +0000 UTC m=+0.341723878 container start b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T18:21:29.738 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:21:29.738 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T18:21:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:21:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:21:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T18:21:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T18:21:29.783 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:21:29.783 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr to start... 2024-03-11T18:21:29.805 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr... 2024-03-11T18:21:29.958 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 bash[31552]: b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 2024-03-11T18:21:29.958 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:29 smithi002 systemd[1]: Started Ceph mgr.x for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:21:30.469 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:30.451+0000 7fa5ff760200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:21:30.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:30 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1953219144' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:21:30.752 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:30.584+0000 7fa5ff760200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:21:31.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:31.072 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:21:31.073 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:21:31.074 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:21:31.075 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:21:31.076 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:21:20.288743+0000", 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:31.077 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.373+0000 7fd707600700 1 Processor -- start 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.373+0000 7fd707600700 1 -- start start 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.373+0000 7fd707600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.373+0000 7fd707600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd700107f60 con 0x7fd700107570 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.374+0000 7fd704c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.374+0000 7fd704c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56284/0 (socket says 172.21.15.2:56284) 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.374+0000 7fd704c00700 1 -- 172.21.15.2:0/1939824218 learned_addr learned my addr 172.21.15.2:0/1939824218 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.374+0000 7fd704c00700 1 -- 172.21.15.2:0/1939824218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd700108720 con 0x7fd700107570 2024-03-11T18:21:31.078 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.375+0000 7fd704c00700 1 --2- 172.21.15.2:0/1939824218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00098a0 tx=0x7fd6f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f806e67039edfe92 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.375+0000 7fd6ffe00700 1 -- 172.21.15.2:0/1939824218 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0004160 con 0x7fd700107570 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.375+0000 7fd6ffe00700 1 -- 172.21.15.2:0/1939824218 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6f00042c0 con 0x7fd700107570 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.375+0000 7fd6ffe00700 1 -- 172.21.15.2:0/1939824218 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0011ac0 con 0x7fd700107570 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 -- 172.21.15.2:0/1939824218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 msgr2=0x7fd700107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 --2- 172.21.15.2:0/1939824218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd6f00098a0 tx=0x7fd6f0009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 -- 172.21.15.2:0/1939824218 shutdown_connections 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 --2- 172.21.15.2:0/1939824218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd700107990 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 -- 172.21.15.2:0/1939824218 >> 172.21.15.2:0/1939824218 conn(0x7fd700102b10 msgr2=0x7fd700104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 -- 172.21.15.2:0/1939824218 shutdown_connections 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.376+0000 7fd707600700 1 -- 172.21.15.2:0/1939824218 wait complete. 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.378+0000 7fd707600700 1 Processor -- start 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.378+0000 7fd707600700 1 -- start start 2024-03-11T18:21:31.079 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.378+0000 7fd707600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.378+0000 7fd707600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70019d9e0 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.379+0000 7fd704c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.379+0000 7fd704c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56298/0 (socket says 172.21.15.2:56298) 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.379+0000 7fd704c00700 1 -- 172.21.15.2:0/1953219144 learned_addr learned my addr 172.21.15.2:0/1953219144 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.379+0000 7fd704c00700 1 -- 172.21.15.2:0/1953219144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f0009550 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.379+0000 7fd704c00700 1 --2- 172.21.15.2:0/1953219144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0000c00 tx=0x7fd6f0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6f0003aa0 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70019dbe0 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6f0003c00 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70019e080 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6f00248b0 con 0x7fd700107570 2024-03-11T18:21:31.080 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd6f002d3f0 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.380+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd6f0019760 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.381+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70010b950 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.384+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd6f0024da0 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.464+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd70010bb60 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.465+0000 7fd6fe000700 1 -- 172.21.15.2:0/1953219144 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd6f001ac60 con 0x7fd700107570 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 msgr2=0x7fd70019d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 --2- 172.21.15.2:0/1953219144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0000c00 tx=0x7fd6f0011b40 comp rx=0 tx=0).stop 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 shutdown_connections 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 --2- 172.21.15.2:0/1953219144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd700107570 0x7fd70019d4a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 >> 172.21.15.2:0/1953219144 conn(0x7fd700102b10 msgr2=0x7fd700103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 shutdown_connections 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:30.467+0000 7fd707600700 1 -- 172.21.15.2:0/1953219144 wait complete. 2024-03-11T18:21:31.081 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (1/15)... 2024-03-11T18:21:31.750 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:31.442+0000 7fa5ff760200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:21:31.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:31.568+0000 7fa5ff760200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:21:31.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:31.749+0000 7fa5ff760200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:21:32.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:31.877+0000 7fa5ff760200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:21:32.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:31.990+0000 7fa5ff760200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:21:32.595 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:32.332+0000 7fa5ff760200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:21:32.596 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:32.476+0000 7fa5ff760200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:21:32.944 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:32.594+0000 7fa5ff760200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:21:32.944 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:32.721+0000 7fa5ff760200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:21:32.944 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:32.943+0000 7fa5ff760200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:21:34.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:33 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1446084400' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:21:34.401 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:34.330+0000 7fa5ff760200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:21:34.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:34.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:21:34.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:21:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:21:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:21:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:21:34.414 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:21:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:21:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:21:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:34.415 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:21:34.417 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:21:34.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:34.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:21:34.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:21:34.420 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:21:34.421 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:21:34.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:21:20.288743+0000", 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:21:34.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.644+0000 7f36d4800700 1 Processor -- start 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.644+0000 7f36d4800700 1 -- start start 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.645+0000 7f36d4800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.645+0000 7f36d4800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc07b8f0 con 0x7f36cc07ce70 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.645+0000 7f36d1e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.645+0000 7f36d1e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56302/0 (socket says 172.21.15.2:56302) 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.645+0000 7f36d1e00700 1 -- 172.21.15.2:0/3356778046 learned_addr learned my addr 172.21.15.2:0/3356778046 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.646+0000 7f36d1e00700 1 -- 172.21.15.2:0/3356778046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc07ba30 con 0x7f36cc07ce70 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.646+0000 7f36d1e00700 1 --2- 172.21.15.2:0/3356778046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f36c00098a0 tx=0x7f36c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c01bc955c13452 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.646+0000 7f36d0a00700 1 -- 172.21.15.2:0/3356778046 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c0004160 con 0x7f36cc07ce70 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.646+0000 7f36d0a00700 1 -- 172.21.15.2:0/3356778046 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c00042c0 con 0x7f36cc07ce70 2024-03-11T18:21:34.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d0a00700 1 -- 172.21.15.2:0/3356778046 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c0011ac0 con 0x7f36cc07ce70 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 -- 172.21.15.2:0/3356778046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 msgr2=0x7f36cc07b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 --2- 172.21.15.2:0/3356778046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f36c00098a0 tx=0x7f36c0009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 -- 172.21.15.2:0/3356778046 shutdown_connections 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 --2- 172.21.15.2:0/3356778046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc07b320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 -- 172.21.15.2:0/3356778046 >> 172.21.15.2:0/3356778046 conn(0x7f36cc102b10 msgr2=0x7f36cc104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 -- 172.21.15.2:0/3356778046 shutdown_connections 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.647+0000 7f36d4800700 1 -- 172.21.15.2:0/3356778046 wait complete. 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.648+0000 7f36d4800700 1 Processor -- start 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d4800700 1 -- start start 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d4800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:34.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d4800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc1a1df0 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d1e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d1e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56308/0 (socket says 172.21.15.2:56308) 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.649+0000 7f36d1e00700 1 -- 172.21.15.2:0/1446084400 learned_addr learned my addr 172.21.15.2:0/1446084400 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36d1e00700 1 -- 172.21.15.2:0/1446084400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009550 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36d1e00700 1 --2- 172.21.15.2:0/1446084400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36c0011a90 tx=0x7f36c0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c0003aa0 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c0003c00 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f36c0024800 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc1a1ff0 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.650+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc1a2490 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.651+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f36c0024cf0 con 0x7f36cc07ce70 2024-03-11T18:21:34.435 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.651+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f36c002d650 con 0x7f36cc07ce70 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.651+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc19b720 con 0x7f36cc07ce70 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.654+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f36c001ab60 con 0x7f36cc07ce70 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.734+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f36cc1097a0 con 0x7f36cc07ce70 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.735+0000 7f36c6a00700 1 -- 172.21.15.2:0/1446084400 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f36c001a430 con 0x7f36cc07ce70 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.736+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 msgr2=0x7f36cc1a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.736+0000 7f36d4800700 1 --2- 172.21.15.2:0/1446084400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36c0011a90 tx=0x7f36c0011b70 comp rx=0 tx=0).stop 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.737+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 shutdown_connections 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.737+0000 7f36d4800700 1 --2- 172.21.15.2:0/1446084400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f36cc07ce70 0x7f36cc1a18b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.737+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 >> 172.21.15.2:0/1446084400 conn(0x7f36cc102b10 msgr2=0x7f36cc198860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.737+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 shutdown_connections 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:33.737+0000 7f36d4800700 1 -- 172.21.15.2:0/1446084400 wait complete. 2024-03-11T18:21:34.436 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (2/15)... 2024-03-11T18:21:34.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:34.512+0000 7fa5ff760200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:21:35.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:34.936+0000 7fa5ff760200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:21:35.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:35.058+0000 7fa5ff760200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:21:35.576 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:35.324+0000 7fa5ff760200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:21:35.957 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:35.574+0000 7fa5ff760200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:21:36.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:35.956+0000 7fa5ff760200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:21:36.586 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:36.279+0000 7fa5ff760200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:21:36.854 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:36.597+0000 7fa5ff760200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:21:37.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:37 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3000332608' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:21:37.815 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:37 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:37.694+0000 7fa5ff760200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:21:37.832 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:37.832 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:37.833 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:21:37.833 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:21:37.834 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:21:37.834 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:21:37.834 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:21:37.834 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.835 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:21:37.835 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:21:37.835 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:21:37.835 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:37.835 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:21:37.836 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:21:37.837 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:21:37.838 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.838 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:21:37.839 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.842 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:21:37.842 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.843 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:21:20.288743+0000", 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:37.844 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:37.845 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:21:37.845 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:37.849 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.091+0000 7f3f7fa00700 1 Processor -- start 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.091+0000 7f3f7fa00700 1 -- start start 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.091+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.091+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7810a190 con 0x7f3f781097a0 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.092+0000 7f3f7d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.092+0000 7f3f7d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48992/0 (socket says 172.21.15.2:48992) 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.092+0000 7f3f7d000700 1 -- 172.21.15.2:0/2048927622 learned_addr learned my addr 172.21.15.2:0/2048927622 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.092+0000 7f3f7d000700 1 -- 172.21.15.2:0/2048927622 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7810a950 con 0x7f3f781097a0 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.093+0000 7f3f7d000700 1 --2- 172.21.15.2:0/2048927622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0098a0 tx=0x7f3f6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f3378e45bf72f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.093+0000 7f3f77400700 1 -- 172.21.15.2:0/2048927622 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f6c004160 con 0x7f3f781097a0 2024-03-11T18:21:37.850 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.093+0000 7f3f77400700 1 -- 172.21.15.2:0/2048927622 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f6c0042c0 con 0x7f3f781097a0 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.093+0000 7f3f77400700 1 -- 172.21.15.2:0/2048927622 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f6c011ac0 con 0x7f3f781097a0 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 -- 172.21.15.2:0/2048927622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 msgr2=0x7f3f78109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 --2- 172.21.15.2:0/2048927622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0098a0 tx=0x7f3f6c009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 -- 172.21.15.2:0/2048927622 shutdown_connections 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 --2- 172.21.15.2:0/2048927622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f78109bc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 -- 172.21.15.2:0/2048927622 >> 172.21.15.2:0/2048927622 conn(0x7f3f7807cd50 msgr2=0x7f3f781074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 -- 172.21.15.2:0/2048927622 shutdown_connections 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.094+0000 7f3f7fa00700 1 -- 172.21.15.2:0/2048927622 wait complete. 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.095+0000 7f3f7fa00700 1 Processor -- start 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.095+0000 7f3f7fa00700 1 -- start start 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.095+0000 7f3f7fa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:37.851 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.095+0000 7f3f7fa00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819da90 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.096+0000 7f3f7d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.096+0000 7f3f7d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49004/0 (socket says 172.21.15.2:49004) 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.096+0000 7f3f7d000700 1 -- 172.21.15.2:0/3000332608 learned_addr learned my addr 172.21.15.2:0/3000332608 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.096+0000 7f3f7d000700 1 -- 172.21.15.2:0/3000332608 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c009550 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.096+0000 7f3f7d000700 1 --2- 172.21.15.2:0/3000332608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c011780 tx=0x7f3f6c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f6c003af0 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f6c003c50 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7819dc90 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f6c0197e0 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7819e130 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3f6c025460 con 0x7f3f781097a0 2024-03-11T18:21:37.852 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.097+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3f6c00bb20 con 0x7f3f781097a0 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.098+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7810db80 con 0x7f3f781097a0 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.100+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3f6c02a060 con 0x7f3f781097a0 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.179+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3f781a0d00 con 0x7f3f781097a0 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.179+0000 7f3f75600700 1 -- 172.21.15.2:0/3000332608 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f3f6c019d60 con 0x7f3f781097a0 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 msgr2=0x7f3f7819d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 --2- 172.21.15.2:0/3000332608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c011780 tx=0x7f3f6c011b40 comp rx=0 tx=0).stop 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 shutdown_connections 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 --2- 172.21.15.2:0/3000332608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f781097a0 0x7f3f7819d550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 >> 172.21.15.2:0/3000332608 conn(0x7f3f7807cd50 msgr2=0x7f3f78106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 shutdown_connections 2024-03-11T18:21:37.853 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:37.181+0000 7f3f7fa00700 1 -- 172.21.15.2:0/3000332608 wait complete. 2024-03-11T18:21:37.854 INFO:teuthology.orchestra.run.smithi002.stdout:mgr not available, waiting (3/15)... 2024-03-11T18:21:38.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:38.416+0000 7fa5ff760200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:21:38.752 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:38.550+0000 7fa5ff760200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:21:39.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:38.846+0000 7fa5ff760200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: Activating manager daemon x 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: mgrmap e2: x(active, starting, since 0.00991613s) 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: Manager daemon x is now available 2024-03-11T18:21:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:21:39.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:21:39.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:38 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' 2024-03-11T18:21:40.169 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:39 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' 2024-03-11T18:21:40.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:39 smithi002 ceph-mon[31328]: from='mgr.14100 172.21.15.2:0/3458993222' entity='mgr.x' 2024-03-11T18:21:40.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:39 smithi002 ceph-mon[31328]: mgrmap e3: x(active, since 1.02072s) 2024-03-11T18:21:41.107 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:40 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3224471292' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 0 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T18:21:41.413 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T18:21:41.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T18:21:41.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T18:21:41.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ], 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T18:21:20.288743+0000", 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout }, 2024-03-11T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.424+0000 7f5c5be00700 1 Processor -- start 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.425+0000 7f5c5be00700 1 -- start start 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.425+0000 7f5c5be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.425+0000 7f5c5be00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c07ae30 con 0x7f5c5c107e60 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.426+0000 7f5c5aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.426+0000 7f5c5aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49080/0 (socket says 172.21.15.2:49080) 2024-03-11T18:21:41.427 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.426+0000 7f5c5aa00700 1 -- 172.21.15.2:0/129551127 learned_addr learned my addr 172.21.15.2:0/129551127 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.426+0000 7f5c5aa00700 1 -- 172.21.15.2:0/129551127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c10a790 con 0x7f5c5c107e60 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.426+0000 7f5c5aa00700 1 --2- 172.21.15.2:0/129551127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098a0 tx=0x7f5c4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18d959a267dbf6b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.427+0000 7f5c59600700 1 -- 172.21.15.2:0/129551127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c004160 con 0x7f5c5c107e60 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.427+0000 7f5c59600700 1 -- 172.21.15.2:0/129551127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4c0042c0 con 0x7f5c5c107e60 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.427+0000 7f5c59600700 1 -- 172.21.15.2:0/129551127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c011ac0 con 0x7f5c5c107e60 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 -- 172.21.15.2:0/129551127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 msgr2=0x7f5c5c10a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 --2- 172.21.15.2:0/129551127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0098a0 tx=0x7f5c4c009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 -- 172.21.15.2:0/129551127 shutdown_connections 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 --2- 172.21.15.2:0/129551127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c10a250 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 -- 172.21.15.2:0/129551127 >> 172.21.15.2:0/129551127 conn(0x7f5c5c101890 msgr2=0x7f5c5c103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 -- 172.21.15.2:0/129551127 shutdown_connections 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.428+0000 7f5c5be00700 1 -- 172.21.15.2:0/129551127 wait complete. 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.429+0000 7f5c5be00700 1 Processor -- start 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.429+0000 7f5c5be00700 1 -- start start 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.429+0000 7f5c5be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.429+0000 7f5c5be00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c19d900 con 0x7f5c5c107e60 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.430+0000 7f5c5aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.430+0000 7f5c5aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49094/0 (socket says 172.21.15.2:49094) 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.430+0000 7f5c5aa00700 1 -- 172.21.15.2:0/3224471292 learned_addr learned my addr 172.21.15.2:0/3224471292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.430+0000 7f5c5aa00700 1 -- 172.21.15.2:0/3224471292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c009550 con 0x7f5c5c107e60 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.430+0000 7f5c5aa00700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c011a70 tx=0x7f5c4c011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:41.429 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.431+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c0039c0 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.431+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4c003b20 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.431+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c19db00 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.431+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c4c024610 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.431+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c19dfa0 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.432+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 3) v1 ==== 45652+0+0 (secure 0 0 0) 0x7f5c4c01aa10 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.432+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c10b590 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.432+0000 7f5c53400700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 0x7f5c4003b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.432+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5c4c04c9c0 con 0x7f5c5c107e60 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.433+0000 7f5c5a000700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 0x7f5c4003b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.433+0000 7f5c5a000700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 0x7f5c4003b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006fd0 tx=0x7f5c48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.439+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5c4c018dc0 con 0x7f5c5c107e60 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.754+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5c5c1a0d70 con 0x7f5c5c107e60 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.755+0000 7f5c53400700 1 -- 172.21.15.2:0/3224471292 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5c4c018be0 con 0x7f5c5c107e60 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.759+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 msgr2=0x7f5c4003b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.759+0000 7f5c5be00700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 0x7f5c4003b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5c48006fd0 tx=0x7f5c48006e40 comp rx=0 tx=0).stop 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.759+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 msgr2=0x7f5c5c19d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.759+0000 7f5c5be00700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c011a70 tx=0x7f5c4c011b50 comp rx=0 tx=0).stop 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 shutdown_connections 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f5c40038b90 0x7f5c4003b050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 --2- 172.21.15.2:0/3224471292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c5c107e60 0x7f5c5c19d3c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:41.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 >> 172.21.15.2:0/3224471292 conn(0x7f5c5c101890 msgr2=0x7f5c5c1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:41.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 shutdown_connections 2024-03-11T18:21:41.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:40.760+0000 7f5c5be00700 1 -- 172.21.15.2:0/3224471292 wait complete. 2024-03-11T18:21:41.432 INFO:teuthology.orchestra.run.smithi002.stdout:mgr is available 2024-03-11T18:21:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:41 smithi002 ceph-mon[31328]: mgrmap e4: x(active, since 2s) 2024-03-11T18:21:43.097 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout fsid = 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T18:21:43.098 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T18:21:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T18:21:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T18:21:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 2024-03-11T18:21:43.099 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.982+0000 7f319a000700 1 Processor -- start 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.983+0000 7f319a000700 1 -- start start 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.983+0000 7f319a000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.983+0000 7f319a000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319407b800 con 0x7f319407ce10 2024-03-11T18:21:43.100 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.983+0000 7f3198c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:43.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.984+0000 7f3198c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49106/0 (socket says 172.21.15.2:49106) 2024-03-11T18:21:43.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.984+0000 7f3198c00700 1 -- 172.21.15.2:0/851331160 learned_addr learned my addr 172.21.15.2:0/851331160 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:43.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.984+0000 7f3198c00700 1 -- 172.21.15.2:0/851331160 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319407b940 con 0x7f319407ce10 2024-03-11T18:21:43.101 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.984+0000 7f3198c00700 1 --2- 172.21.15.2:0/851331160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f31780098a0 tx=0x7f3178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1889bc8b96fcf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.985+0000 7f3193e00700 1 -- 172.21.15.2:0/851331160 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3178004160 con 0x7f319407ce10 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.985+0000 7f3193e00700 1 -- 172.21.15.2:0/851331160 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31780042c0 con 0x7f319407ce10 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.985+0000 7f3193e00700 1 -- 172.21.15.2:0/851331160 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3178011ac0 con 0x7f319407ce10 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.985+0000 7f319a000700 1 -- 172.21.15.2:0/851331160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 msgr2=0x7f319407b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.985+0000 7f319a000700 1 --2- 172.21.15.2:0/851331160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f31780098a0 tx=0x7f3178009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.986+0000 7f319a000700 1 -- 172.21.15.2:0/851331160 shutdown_connections 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.986+0000 7f319a000700 1 --2- 172.21.15.2:0/851331160 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319407b2c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.986+0000 7f319a000700 1 -- 172.21.15.2:0/851331160 >> 172.21.15.2:0/851331160 conn(0x7f31941028d0 msgr2=0x7f3194104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.986+0000 7f319a000700 1 -- 172.21.15.2:0/851331160 shutdown_connections 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.986+0000 7f319a000700 1 -- 172.21.15.2:0/851331160 wait complete. 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.987+0000 7f319a000700 1 Processor -- start 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.987+0000 7f319a000700 1 -- start start 2024-03-11T18:21:43.102 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.987+0000 7f319a000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.987+0000 7f319a000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319419d8b0 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.988+0000 7f3198c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.988+0000 7f3198c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49120/0 (socket says 172.21.15.2:49120) 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.988+0000 7f3198c00700 1 -- 172.21.15.2:0/2485141372 learned_addr learned my addr 172.21.15.2:0/2485141372 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.988+0000 7f3198c00700 1 -- 172.21.15.2:0/2485141372 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178009550 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.988+0000 7f3198c00700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3178011a70 tx=0x7f3178011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.989+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31780039c0 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.989+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3178003b20 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.989+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319419dab0 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.989+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3178024610 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.989+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319419df50 con 0x7f319407ce10 2024-03-11T18:21:43.103 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.990+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f317801a430 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.990+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319410d700 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.990+0000 7f3192000700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 0x7f317003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.990+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f317804cc00 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.991+0000 7f318be00700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 0x7f317003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.992+0000 7f318be00700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 0x7f317003b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:41.996+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3178010380 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.218+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f31941a09c0 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.219+0000 7f3192000700 1 -- 172.21.15.2:0/2485141372 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f317804ce60 con 0x7f319407ce10 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 msgr2=0x7f317003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 0x7f317003b170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180006e40 comp rx=0 tx=0).stop 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 msgr2=0x7f319419d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3178011a70 tx=0x7f3178011b50 comp rx=0 tx=0).stop 2024-03-11T18:21:43.104 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 shutdown_connections 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f3170038cb0 0x7f317003b170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 --2- 172.21.15.2:0/2485141372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f319407ce10 0x7f319419d370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 >> 172.21.15.2:0/2485141372 conn(0x7f31941028d0 msgr2=0x7f3194103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 shutdown_connections 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:42.223+0000 7f319a000700 1 -- 172.21.15.2:0/2485141372 wait complete. 2024-03-11T18:21:43.105 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling cephadm module... 2024-03-11T18:21:43.248 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:42 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2485141372' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T18:21:44.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2468099605' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T18:21:44.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setuser ceph since I am not root 2024-03-11T18:21:44.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setgroup ceph since I am not root 2024-03-11T18:21:44.706 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.677+0000 7f83da400700 1 Processor -- start 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.677+0000 7f83da400700 1 -- start start 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.678+0000 7f83da400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.678+0000 7f83da400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d4107de0 con 0x7f83d41073f0 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.678+0000 7f83d9000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.678+0000 7f83d9000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49122/0 (socket says 172.21.15.2:49122) 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.678+0000 7f83d9000700 1 -- 172.21.15.2:0/425208413 learned_addr learned my addr 172.21.15.2:0/425208413 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.679+0000 7f83d9000700 1 -- 172.21.15.2:0/425208413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d4108550 con 0x7f83d41073f0 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.679+0000 7f83d9000700 1 --2- 172.21.15.2:0/425208413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f83c80098a0 tx=0x7f83c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522c67a5f9e142df server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:44.707 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.680+0000 7f83d3400700 1 -- 172.21.15.2:0/425208413 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83c8004160 con 0x7f83d41073f0 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.680+0000 7f83d3400700 1 -- 172.21.15.2:0/425208413 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c80042c0 con 0x7f83d41073f0 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.680+0000 7f83d3400700 1 -- 172.21.15.2:0/425208413 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83c8011ac0 con 0x7f83d41073f0 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.680+0000 7f83da400700 1 -- 172.21.15.2:0/425208413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 msgr2=0x7f83d4107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.680+0000 7f83da400700 1 --2- 172.21.15.2:0/425208413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f83c80098a0 tx=0x7f83c8009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.681+0000 7f83da400700 1 -- 172.21.15.2:0/425208413 shutdown_connections 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.681+0000 7f83da400700 1 --2- 172.21.15.2:0/425208413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d4107810 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.681+0000 7f83da400700 1 -- 172.21.15.2:0/425208413 >> 172.21.15.2:0/425208413 conn(0x7f83d41029b0 msgr2=0x7f83d4104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.681+0000 7f83da400700 1 -- 172.21.15.2:0/425208413 shutdown_connections 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.681+0000 7f83da400700 1 -- 172.21.15.2:0/425208413 wait complete. 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.682+0000 7f83da400700 1 Processor -- start 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.682+0000 7f83da400700 1 -- start start 2024-03-11T18:21:44.708 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.682+0000 7f83da400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.682+0000 7f83da400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d407d090 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.683+0000 7f83d9000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.683+0000 7f83d9000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49124/0 (socket says 172.21.15.2:49124) 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.683+0000 7f83d9000700 1 -- 172.21.15.2:0/2468099605 learned_addr learned my addr 172.21.15.2:0/2468099605 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.683+0000 7f83d9000700 1 -- 172.21.15.2:0/2468099605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c8009550 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83d9000700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009fb0 tx=0x7f83c8011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83c80038b0 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83c8003a10 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f83c80195e0 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d407b280 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.684+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d407b720 con 0x7f83d41073f0 2024-03-11T18:21:44.709 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.685+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 45758+0+0 (secure 0 0 0) 0x7f83c801aa10 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.685+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d410b780 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.686+0000 7f83d1600700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 0x7f83bc03b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.686+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f83c804cd30 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.686+0000 7f83d3e00700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 0x7f83bc03b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.687+0000 7f83d3e00700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 0x7f83bc03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f83c0006fd0 tx=0x7f83c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.691+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83c8003b80 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:43.969+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f83d410b980 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.022+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f83c8023c20 con 0x7f83d41073f0 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.025+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 msgr2=0x7f83bc03b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.025+0000 7f83da400700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 0x7f83bc03b120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f83c0006fd0 tx=0x7f83c0006e40 comp rx=0 tx=0).stop 2024-03-11T18:21:44.710 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.025+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 msgr2=0x7f83d407cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.025+0000 7f83d1600700 1 -- 172.21.15.2:0/2468099605 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f83c8023390 con 0x7f83d41073f0 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.025+0000 7f83da400700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009fb0 tx=0x7f83c8011b20 comp rx=0 tx=0).stop 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 shutdown_connections 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f83bc038c60 0x7f83bc03b120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 --2- 172.21.15.2:0/2468099605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83d41073f0 0x7f83d407cb50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 >> 172.21.15.2:0/2468099605 conn(0x7f83d41029b0 msgr2=0x7f83d4103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 shutdown_connections 2024-03-11T18:21:44.711 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:44.027+0000 7f83da400700 1 -- 172.21.15.2:0/2468099605 wait complete. 2024-03-11T18:21:44.943 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:44.812+0000 7f87913b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:21:44.943 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:44.941+0000 7f87913b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:21:45.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2468099605' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T18:21:45.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:45 smithi002 ceph-mon[31328]: mgrmap e5: x(active, since 5s) 2024-03-11T18:21:45.883 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:45 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:45.764+0000 7f87913b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:21:46.181 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/736459244' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T18:21:46.181 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:45 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:45.894+0000 7f87913b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:21:46.181 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.076+0000 7f87913b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:21:46.189 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.230+0000 7f26f9c00700 1 Processor -- start 2024-03-11T18:21:46.191 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.230+0000 7f26f9c00700 1 -- start start 2024-03-11T18:21:46.192 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.231+0000 7f26f9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:46.192 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.231+0000 7f26f9c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f407ae50 con 0x7f26f4105e10 2024-03-11T18:21:46.192 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.231+0000 7f26f2a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.231+0000 7f26f2a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49142/0 (socket says 172.21.15.2:49142) 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.231+0000 7f26f2a00700 1 -- 172.21.15.2:0/3290825404 learned_addr learned my addr 172.21.15.2:0/3290825404 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.232+0000 7f26f2a00700 1 -- 172.21.15.2:0/3290825404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4106770 con 0x7f26f4105e10 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.232+0000 7f26f2a00700 1 --2- 172.21.15.2:0/3290825404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26e40098a0 tx=0x7f26e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e6febde9222b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.232+0000 7f26f1600700 1 -- 172.21.15.2:0/3290825404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26e4004160 con 0x7f26f4105e10 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f1600700 1 -- 172.21.15.2:0/3290825404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26e40042c0 con 0x7f26f4105e10 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f1600700 1 -- 172.21.15.2:0/3290825404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26e4011ac0 con 0x7f26f4105e10 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 -- 172.21.15.2:0/3290825404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 msgr2=0x7f26f4106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:46.198 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 --2- 172.21.15.2:0/3290825404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26e40098a0 tx=0x7f26e4009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 -- 172.21.15.2:0/3290825404 shutdown_connections 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 --2- 172.21.15.2:0/3290825404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f4106230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 -- 172.21.15.2:0/3290825404 >> 172.21.15.2:0/3290825404 conn(0x7f26f4101a50 msgr2=0x7f26f4103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 -- 172.21.15.2:0/3290825404 shutdown_connections 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.233+0000 7f26f9c00700 1 -- 172.21.15.2:0/3290825404 wait complete. 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.235+0000 7f26f9c00700 1 Processor -- start 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.235+0000 7f26f9c00700 1 -- start start 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.235+0000 7f26f9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.235+0000 7f26f9c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f419da80 con 0x7f26f4105e10 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.236+0000 7f26f2a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.236+0000 7f26f2a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49154/0 (socket says 172.21.15.2:49154) 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.236+0000 7f26f2a00700 1 -- 172.21.15.2:0/736459244 learned_addr learned my addr 172.21.15.2:0/736459244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:46.199 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.236+0000 7f26f2a00700 1 -- 172.21.15.2:0/736459244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4009550 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.236+0000 7f26f2a00700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011a70 tx=0x7f26e4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.237+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26e40039c0 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.237+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26e4003b20 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.237+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26e4024610 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.237+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f419dc80 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.237+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f419e120 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.238+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f41971e0 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.239+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f26e401aa10 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.239+0000 7f26eb400700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 0x7f26d803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.239+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f26e404cd20 con 0x7f26f4105e10 2024-03-11T18:21:46.200 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.239+0000 7f26f2000700 1 -- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 msgr2=0x7f26d803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.239+0000 7f26f2000700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 0x7f26d803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.245+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f26e4018dc0 con 0x7f26f4105e10 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.440+0000 7f26f2000700 1 -- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 msgr2=0x7f26d803b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.440+0000 7f26f2000700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 0x7f26d803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.522+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f26f402d8e0 con 0x7f26f4105e10 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.523+0000 7f26eb400700 1 -- 172.21.15.2:0/736459244 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f26e4018be0 con 0x7f26f4105e10 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 msgr2=0x7f26d803b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 0x7f26d803b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 msgr2=0x7f26f419d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26e4011a70 tx=0x7f26e4011b50 comp rx=0 tx=0).stop 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 shutdown_connections 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f26d8038d00 0x7f26d803b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:46.201 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 --2- 172.21.15.2:0/736459244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26f4105e10 0x7f26f419d540 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:46.202 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.525+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 >> 172.21.15.2:0/736459244 conn(0x7f26f4101a50 msgr2=0x7f26f4194320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:46.202 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.526+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 shutdown_connections 2024-03-11T18:21:46.202 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:45.526+0000 7f26f9c00700 1 -- 172.21.15.2:0/736459244 wait complete. 2024-03-11T18:21:46.202 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-03-11T18:21:46.202 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 5... 2024-03-11T18:21:46.446 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.211+0000 7f87913b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:21:46.446 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.327+0000 7f87913b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:21:46.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.678+0000 7f87913b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:21:47.069 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.824+0000 7f87913b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:21:47.069 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:46.940+0000 7f87913b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:21:47.069 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:47 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:47.066+0000 7f87913b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:21:47.367 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:47 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:47.290+0000 7f87913b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:21:49.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:48 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:48.605+0000 7f87913b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:21:49.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:48 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:48.781+0000 7f87913b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:21:49.501 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:49 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:49.208+0000 7f87913b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:21:49.501 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:49 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:49.331+0000 7f87913b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:21:49.850 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:49 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:49.595+0000 7f87913b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:21:50.235 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:49 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:49.848+0000 7f87913b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:21:50.501 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:50 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:50.233+0000 7f87913b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:21:50.872 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:50 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:50.557+0000 7f87913b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:21:51.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:50 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:50.870+0000 7f87913b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:21:52.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:51 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:51.924+0000 7f87913b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:21:52.973 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:52 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:52.638+0000 7f87913b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:21:52.974 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:52 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:52.771+0000 7f87913b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:21:53.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:21:53.060+0000 7f87913b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: Active manager daemon x restarted 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: Activating manager daemon x 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: mgrmap e6: x(active, starting, since 0.00733071s) 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:21:53.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:53 smithi002 ceph-mon[31328]: Manager daemon x is now available 2024-03-11T18:21:54.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: Found migration_current of "None". Setting to last migration. 2024-03-11T18:21:54.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:21:54.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:21:54.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:21:54.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:21:54.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: mgrmap e7: x(active, since 1.01671s) 2024-03-11T18:21:54.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:54 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:21:54.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:21:54.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T18:21:54.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T18:21:54.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:21:54.804 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.755+0000 7f71be800700 1 Processor -- start 2024-03-11T18:21:54.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.755+0000 7f71be800700 1 -- start start 2024-03-11T18:21:54.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.756+0000 7f71be800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:54.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.756+0000 7f71be800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b807ae10 con 0x7f71b8105e50 2024-03-11T18:21:54.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.756+0000 7f71b7400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:54.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.756+0000 7f71b7400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49156/0 (socket says 172.21.15.2:49156) 2024-03-11T18:21:54.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.756+0000 7f71b7400700 1 -- 172.21.15.2:0/3161460620 learned_addr learned my addr 172.21.15.2:0/3161460620 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:54.812 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.757+0000 7f71b7400700 1 -- 172.21.15.2:0/3161460620 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b81067b0 con 0x7f71b8105e50 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.757+0000 7f71b7400700 1 --2- 172.21.15.2:0/3161460620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71a80098a0 tx=0x7f71a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f711f71f9cacedb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.757+0000 7f71b6000700 1 -- 172.21.15.2:0/3161460620 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71a8004160 con 0x7f71b8105e50 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.757+0000 7f71b6000700 1 -- 172.21.15.2:0/3161460620 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71a80042c0 con 0x7f71b8105e50 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.757+0000 7f71b6000700 1 -- 172.21.15.2:0/3161460620 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71a8011ac0 con 0x7f71b8105e50 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 -- 172.21.15.2:0/3161460620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 msgr2=0x7f71b8106270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 --2- 172.21.15.2:0/3161460620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71a80098a0 tx=0x7f71a8009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 -- 172.21.15.2:0/3161460620 shutdown_connections 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 --2- 172.21.15.2:0/3161460620 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b8106270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 -- 172.21.15.2:0/3161460620 >> 172.21.15.2:0/3161460620 conn(0x7f71b8101a50 msgr2=0x7f71b8103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 -- 172.21.15.2:0/3161460620 shutdown_connections 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.758+0000 7f71be800700 1 -- 172.21.15.2:0/3161460620 wait complete. 2024-03-11T18:21:54.813 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71be800700 1 Processor -- start 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71be800700 1 -- start start 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71be800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71be800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b819da80 con 0x7f71b8105e50 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71b7400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71b7400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49164/0 (socket says 172.21.15.2:49164) 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.760+0000 7f71b7400700 1 -- 172.21.15.2:0/2094640605 learned_addr learned my addr 172.21.15.2:0/2094640605 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71b7400700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a8009550 con 0x7f71b8105e50 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71b7400700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f71a8011a70 tx=0x7f71a8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71a80039c0 con 0x7f71b8105e50 2024-03-11T18:21:54.814 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71a8003b20 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b819dc80 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71a8024610 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.761+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b819e120 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f71a801aa10 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71afe00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f719c03b8d0 con 0x7f719c038d00 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f71a804cc40 con 0x7f71b8105e50 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.763+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.964+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:46.964+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:47.365+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.815 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:47.365+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:48.166+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:48.166+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:49.768+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:49.768+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:52.971+0000 7f71b6a00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1779941333 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:52.971+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:53.069+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 6) v1 ==== 45536+0+0 (secure 0 0 0) 0x7f71a802d910 con 0x7f71b8105e50 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:53.070+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 msgr2=0x7f719c03b1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:53.070+0000 7f71afe00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.074+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 45663+0+0 (secure 0 0 0) 0x7f71a804de90 con 0x7f71b8105e50 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.074+0000 7f71afe00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 0x7f719c03e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.075+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f719c03b8d0 con 0x7f719c03c5f0 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.076+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 0x7f719c03e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:54.816 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.079+0000 7f71b6a00700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 0x7f719c03e9e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71a4003a10 tx=0x7f71a40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.082+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f719c03b8d0 con 0x7f719c03c5f0 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.091+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f71b8107680 con 0x7f719c03c5f0 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.092+0000 7f71afe00700 1 -- 172.21.15.2:0/2094640605 <== mgr.14120 v2:172.21.15.2:6800/2981412802 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f71b8107680 con 0x7f719c03c5f0 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.092+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 msgr2=0x7f719c03e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.092+0000 7f71be800700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 0x7f719c03e9e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f71a4003a10 tx=0x7f71a40092b0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 msgr2=0x7f71b819d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f71a8011a70 tx=0x7f71a8011b50 comp rx=0 tx=0).stop 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 shutdown_connections 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f719c03c5f0 0x7f719c03e9e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71b8105e50 0x7f71b819d540 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 --2- 172.21.15.2:0/2094640605 >> [v2:172.21.15.2:6800/1779941333,v1:172.21.15.2:6801/1779941333] conn(0x7f719c038d00 0x7f719c03b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 >> 172.21.15.2:0/2094640605 conn(0x7f71b8101a50 msgr2=0x7f71b8194310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 shutdown_connections 2024-03-11T18:21:54.817 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:54.093+0000 7f71be800700 1 -- 172.21.15.2:0/2094640605 wait complete. 2024-03-11T18:21:54.818 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 5 is available 2024-03-11T18:21:54.818 INFO:teuthology.orchestra.run.smithi002.stdout:Setting orchestrator backend to cephadm... 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: [11/Mar/2024:18:21:53] ENGINE Bus STARTING 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: [11/Mar/2024:18:21:53] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: [11/Mar/2024:18:21:54] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: [11/Mar/2024:18:21:54] ENGINE Bus STARTED 2024-03-11T18:21:55.394 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:55 smithi002 ceph-mon[31328]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T18:21:56.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:56 smithi002 ceph-mon[31328]: mgrmap e8: x(active, since 2s) 2024-03-11T18:21:56.294 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:56 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:21:56.294 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:56 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:21:56.294 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.391+0000 7f2071400700 1 Processor -- start 2024-03-11T18:21:56.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.392+0000 7f2071400700 1 -- start start 2024-03-11T18:21:56.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.392+0000 7f2071400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:56.295 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.392+0000 7f2071400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c10a190 con 0x7f206c1097a0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.393+0000 7f206a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.393+0000 7f206a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49786/0 (socket says 172.21.15.2:49786) 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.393+0000 7f206a800700 1 -- 172.21.15.2:0/4056149510 learned_addr learned my addr 172.21.15.2:0/4056149510 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.393+0000 7f206a800700 1 -- 172.21.15.2:0/4056149510 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c10a950 con 0x7f206c1097a0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.393+0000 7f206a800700 1 --2- 172.21.15.2:0/4056149510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688a22dfe7c8683a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.394+0000 7f2069e00700 1 -- 172.21.15.2:0/4056149510 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f205c004160 con 0x7f206c1097a0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.394+0000 7f2069e00700 1 -- 172.21.15.2:0/4056149510 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f205c0042c0 con 0x7f206c1097a0 2024-03-11T18:21:56.302 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.394+0000 7f2069e00700 1 -- 172.21.15.2:0/4056149510 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f205c011ac0 con 0x7f206c1097a0 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 -- 172.21.15.2:0/4056149510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 msgr2=0x7f206c109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 --2- 172.21.15.2:0/4056149510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 -- 172.21.15.2:0/4056149510 shutdown_connections 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 --2- 172.21.15.2:0/4056149510 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c109bc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 -- 172.21.15.2:0/4056149510 >> 172.21.15.2:0/4056149510 conn(0x7f206c07cdf0 msgr2=0x7f206c1074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 -- 172.21.15.2:0/4056149510 shutdown_connections 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.395+0000 7f2071400700 1 -- 172.21.15.2:0/4056149510 wait complete. 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.396+0000 7f2071400700 1 Processor -- start 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.396+0000 7f2071400700 1 -- start start 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.397+0000 7f2071400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:56.303 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.397+0000 7f2071400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206c19dad0 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.397+0000 7f206a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.397+0000 7f206a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49802/0 (socket says 172.21.15.2:49802) 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.397+0000 7f206a800700 1 -- 172.21.15.2:0/231842736 learned_addr learned my addr 172.21.15.2:0/231842736 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f206a800700 1 -- 172.21.15.2:0/231842736 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c009550 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f206a800700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f205c011780 tx=0x7f205c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f205c003af0 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f205c003c50 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f2071400700 1 -- 172.21.15.2:0/231842736 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c19dcd0 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f205c0197e0 con 0x7f206c1097a0 2024-03-11T18:21:56.304 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.398+0000 7f2071400700 1 -- 172.21.15.2:0/231842736 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c19e170 con 0x7f206c1097a0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.399+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f205c01aba0 con 0x7f206c1097a0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.400+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2044005330 con 0x7f206c1097a0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.400+0000 7f2063e00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 0x7f205003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.400+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f205c04d010 con 0x7f206c1097a0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.400+0000 7f2061e00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 0x7f205003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.401+0000 7f2061e00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 0x7f205003b1c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f205400ad30 tx=0x7f20540093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.406+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f205c01a430 con 0x7f206c1097a0 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.648+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2044000bc0 con 0x7f2050038d00 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.662+0000 7f2063e00700 1 -- 172.21.15.2:0/231842736 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2044000bc0 con 0x7f2050038d00 2024-03-11T18:21:56.305 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 msgr2=0x7f205003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 0x7f205003b1c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f205400ad30 tx=0x7f20540093f0 comp rx=0 tx=0).stop 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 msgr2=0x7f206c19d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f205c011780 tx=0x7f205c011b40 comp rx=0 tx=0).stop 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 shutdown_connections 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f2050038d00 0x7f205003b1c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 --2- 172.21.15.2:0/231842736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f206c1097a0 0x7f206c19d590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 >> 172.21.15.2:0/231842736 conn(0x7f206c07cdf0 msgr2=0x7f206c106c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.665+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 shutdown_connections 2024-03-11T18:21:56.306 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:55.666+0000 7f2060a00700 1 -- 172.21.15.2:0/231842736 wait complete. 2024-03-11T18:21:57.496 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:57 smithi002 ceph-mon[31328]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:21:57.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T18:21:57.805 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.877+0000 7f4316600700 1 Processor -- start 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.877+0000 7f4316600700 1 -- start start 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.878+0000 7f4316600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.878+0000 7f4316600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431007b800 con 0x7f431007ce10 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.878+0000 7f4315200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.878+0000 7f4315200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49818/0 (socket says 172.21.15.2:49818) 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.878+0000 7f4315200700 1 -- 172.21.15.2:0/2913985138 learned_addr learned my addr 172.21.15.2:0/2913985138 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.879+0000 7f4315200700 1 -- 172.21.15.2:0/2913985138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431007b940 con 0x7f431007ce10 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.879+0000 7f4315200700 1 --2- 172.21.15.2:0/2913985138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f43040098a0 tx=0x7f4304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c988d02b4653fa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:57.806 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.879+0000 7f430f400700 1 -- 172.21.15.2:0/2913985138 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4304004160 con 0x7f431007ce10 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.879+0000 7f430f400700 1 -- 172.21.15.2:0/2913985138 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43040042c0 con 0x7f431007ce10 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.879+0000 7f430f400700 1 -- 172.21.15.2:0/2913985138 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4304011ac0 con 0x7f431007ce10 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 -- 172.21.15.2:0/2913985138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 msgr2=0x7f431007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 --2- 172.21.15.2:0/2913985138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f43040098a0 tx=0x7f4304009bb0 comp rx=0 tx=0).stop 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 -- 172.21.15.2:0/2913985138 shutdown_connections 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 --2- 172.21.15.2:0/2913985138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431007b2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 -- 172.21.15.2:0/2913985138 >> 172.21.15.2:0/2913985138 conn(0x7f43101028d0 msgr2=0x7f4310104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 -- 172.21.15.2:0/2913985138 shutdown_connections 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.880+0000 7f4316600700 1 -- 172.21.15.2:0/2913985138 wait complete. 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.882+0000 7f4316600700 1 Processor -- start 2024-03-11T18:21:57.807 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.882+0000 7f4316600700 1 -- start start 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4316600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4316600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431019d8b0 con 0x7f431007ce10 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4315200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4315200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49826/0 (socket says 172.21.15.2:49826) 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4315200700 1 -- 172.21.15.2:0/2201569721 learned_addr learned my addr 172.21.15.2:0/2201569721 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.883+0000 7f4315200700 1 -- 172.21.15.2:0/2201569721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4304009550 con 0x7f431007ce10 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f4315200700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4304009fb0 tx=0x7f4304011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43040038b0 con 0x7f431007ce10 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4304003a10 con 0x7f431007ce10 2024-03-11T18:21:57.808 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43040195e0 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431019dab0 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.884+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431019df50 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.885+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f430401a430 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.886+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431010d700 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.886+0000 7f430d600700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 0x7f42f803b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.886+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f430404cef0 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.887+0000 7f430fe00700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 0x7f42f803b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.887+0000 7f430fe00700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 0x7f42f803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4300006fd0 tx=0x7f4300006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:56.891+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4304003b80 con 0x7f431007ce10 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.120+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4310103ba0 con 0x7f42f8038cb0 2024-03-11T18:21:57.809 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.121+0000 7f430d600700 1 -- 172.21.15.2:0/2201569721 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4310103ba0 con 0x7f42f8038cb0 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 msgr2=0x7f42f803b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 0x7f42f803b170 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4300006fd0 tx=0x7f4300006e40 comp rx=0 tx=0).stop 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 msgr2=0x7f431019d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4304009fb0 tx=0x7f4304011b20 comp rx=0 tx=0).stop 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 shutdown_connections 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f42f8038cb0 0x7f42f803b170 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 --2- 172.21.15.2:0/2201569721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f431007ce10 0x7f431019d370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 >> 172.21.15.2:0/2201569721 conn(0x7f43101028d0 msgr2=0x7f4310103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 shutdown_connections 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:57.124+0000 7f4316600700 1 -- 172.21.15.2:0/2201569721 wait complete. 2024-03-11T18:21:57.810 INFO:teuthology.orchestra.run.smithi002.stdout:Generating ssh key... 2024-03-11T18:21:58.360 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-mon[31328]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:21:59.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: Generating public/private rsa key pair. 2024-03-11T18:21:59.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: Your identification has been saved in /tmp/tmp6syv2x8b/key. 2024-03-11T18:21:59.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: Your public key has been saved in /tmp/tmp6syv2x8b/key.pub. 2024-03-11T18:21:59.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: The key fingerprint is: 2024-03-11T18:21:59.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: SHA256:Pe/sZHOuslo4N1nsJN2mbqTglFzx5nHHYp0MtxFppRc ceph-02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: The key's randomart image is: 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: +---[RSA 3072]----+ 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | E=| 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | . . *.| 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | o *o=| 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | ..o+.+=+| 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | .So+o=+oo | 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | =. Ooo | 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | oo.=oB . | 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | .+oB.+ | 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: | ...=*.. | 2024-03-11T18:21:59.254 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:21:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: +----[SHA256]-----+ 2024-03-11T18:21:59.717 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.432+0000 7fa8cda00700 1 Processor -- start 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.433+0000 7fa8cda00700 1 -- start start 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.433+0000 7fa8cda00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.433+0000 7fa8cda00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c807b8d0 con 0x7fa8c807ce50 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.434+0000 7fa8c6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.434+0000 7fa8c6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42186/0 (socket says 172.21.15.2:42186) 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.434+0000 7fa8c6a00700 1 -- 172.21.15.2:0/2735406060 learned_addr learned my addr 172.21.15.2:0/2735406060 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.434+0000 7fa8c6a00700 1 -- 172.21.15.2:0/2735406060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c807ba10 con 0x7fa8c807ce50 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8c6a00700 1 --2- 172.21.15.2:0/2735406060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e80 tx=0x7fa8b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6e93d84da0f216 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8c5600700 1 -- 172.21.15.2:0/2735406060 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8b800e070 con 0x7fa8c807ce50 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8c5600700 1 -- 172.21.15.2:0/2735406060 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8b800b8c0 con 0x7fa8c807ce50 2024-03-11T18:21:59.718 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8c5600700 1 -- 172.21.15.2:0/2735406060 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8b8004550 con 0x7fa8c807ce50 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8cda00700 1 -- 172.21.15.2:0/2735406060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 msgr2=0x7fa8c807b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.435+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2735406060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009e80 tx=0x7fa8b800b330 comp rx=0 tx=0).stop 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.436+0000 7fa8cda00700 1 -- 172.21.15.2:0/2735406060 shutdown_connections 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.436+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2735406060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c807b300 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.436+0000 7fa8cda00700 1 -- 172.21.15.2:0/2735406060 >> 172.21.15.2:0/2735406060 conn(0x7fa8c8102b10 msgr2=0x7fa8c8104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.436+0000 7fa8cda00700 1 -- 172.21.15.2:0/2735406060 shutdown_connections 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.436+0000 7fa8cda00700 1 -- 172.21.15.2:0/2735406060 wait complete. 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8cda00700 1 Processor -- start 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8cda00700 1 -- start start 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8cda00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8cda00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c819d8f0 con 0x7fa8c807ce50 2024-03-11T18:21:59.719 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8c6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8c6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42198/0 (socket says 172.21.15.2:42198) 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.437+0000 7fa8c6a00700 1 -- 172.21.15.2:0/2075471136 learned_addr learned my addr 172.21.15.2:0/2075471136 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8c6a00700 1 -- 172.21.15.2:0/2075471136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009b30 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8c6a00700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009fd0 tx=0x7fa8b8003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8b800e040 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8b80128f0 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c819daf0 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.438+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8b801c890 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.439+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c819df70 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.440+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fa8b8012a60 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.440+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c810d920 con 0x7fa8c807ce50 2024-03-11T18:21:59.720 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.440+0000 7fa8bf400700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 0x7fa8ac03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.440+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa8b804cda0 con 0x7fa8c807ce50 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.440+0000 7fa8c6000700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 0x7fa8ac03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.441+0000 7fa8c6000700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 0x7fa8ac03b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4006fd0 tx=0x7fa8b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.446+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa8b8024c00 con 0x7fa8c807ce50 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.672+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8c8103c00 con 0x7fa8ac038d60 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.995+0000 7fa8bf400700 1 -- 172.21.15.2:0/2075471136 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa8c8103c00 con 0x7fa8ac038d60 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.998+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 msgr2=0x7fa8ac03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.998+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 0x7fa8ac03b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4006fd0 tx=0x7fa8b4006e40 comp rx=0 tx=0).stop 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.998+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 msgr2=0x7fa8c819d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.998+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009fd0 tx=0x7fa8b8003d70 comp rx=0 tx=0).stop 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 shutdown_connections 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa8ac038d60 0x7fa8ac03b220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:59.721 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 --2- 172.21.15.2:0/2075471136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8c807ce50 0x7fa8c819d3b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:21:59.722 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 >> 172.21.15.2:0/2075471136 conn(0x7fa8c8102b10 msgr2=0x7fa8c81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:21:59.722 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 shutdown_connections 2024-03-11T18:21:59.722 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:21:58.999+0000 7fa8cda00700 1 -- 172.21.15.2:0/2075471136 wait complete. 2024-03-11T18:22:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:59 smithi002 ceph-mon[31328]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:59 smithi002 ceph-mon[31328]: Generating ssh key... 2024-03-11T18:22:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:59 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:21:59 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:01.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.303+0000 7f4d5b200700 1 Processor -- start 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.303+0000 7f4d5b200700 1 -- start start 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.303+0000 7f4d5b200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.303+0000 7f4d5b200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5410a160 con 0x7f4d54109770 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.304+0000 7f4d53e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.304+0000 7f4d53e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42212/0 (socket says 172.21.15.2:42212) 2024-03-11T18:22:01.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.304+0000 7f4d53e00700 1 -- 172.21.15.2:0/2196464736 learned_addr learned my addr 172.21.15.2:0/2196464736 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:01.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.304+0000 7f4d53e00700 1 -- 172.21.15.2:0/2196464736 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5410a920 con 0x7f4d54109770 2024-03-11T18:22:01.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.304+0000 7f4d53e00700 1 --2- 172.21.15.2:0/2196464736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d440098a0 tx=0x7f4d44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab501f370524e7d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:01.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d53400700 1 -- 172.21.15.2:0/2196464736 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d44004160 con 0x7f4d54109770 2024-03-11T18:22:01.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d53400700 1 -- 172.21.15.2:0/2196464736 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d440042c0 con 0x7f4d54109770 2024-03-11T18:22:01.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d53400700 1 -- 172.21.15.2:0/2196464736 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d44011ac0 con 0x7f4d54109770 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d5b200700 1 -- 172.21.15.2:0/2196464736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 msgr2=0x7f4d54109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d5b200700 1 --2- 172.21.15.2:0/2196464736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d440098a0 tx=0x7f4d44009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d5b200700 1 -- 172.21.15.2:0/2196464736 shutdown_connections 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d5b200700 1 --2- 172.21.15.2:0/2196464736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d54109b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.305+0000 7f4d5b200700 1 -- 172.21.15.2:0/2196464736 >> 172.21.15.2:0/2196464736 conn(0x7f4d5407cba0 msgr2=0x7f4d541074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.306+0000 7f4d5b200700 1 -- 172.21.15.2:0/2196464736 shutdown_connections 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.306+0000 7f4d5b200700 1 -- 172.21.15.2:0/2196464736 wait complete. 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d5b200700 1 Processor -- start 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d5b200700 1 -- start start 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d5b200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d5b200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5419da80 con 0x7f4d54109770 2024-03-11T18:22:01.221 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d53e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d53e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42228/0 (socket says 172.21.15.2:42228) 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.307+0000 7f4d53e00700 1 -- 172.21.15.2:0/3600612 learned_addr learned my addr 172.21.15.2:0/3600612 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d53e00700 1 -- 172.21.15.2:0/3600612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44009550 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d53e00700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000c00 tx=0x7f4d44011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d44003aa0 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d44003c00 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5419dc80 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.308+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d440248a0 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.309+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5419e120 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.310+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f4d4401ac60 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.310+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5410db50 con 0x7f4d54109770 2024-03-11T18:22:01.222 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.310+0000 7f4d51600700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 0x7f4d3803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.310+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d4404cd80 con 0x7f4d54109770 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.310+0000 7f4d4b400700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 0x7f4d3803b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.311+0000 7f4d4b400700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 0x7f4d3803b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c006fd0 tx=0x7f4d3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.316+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4d44018530 con 0x7f4d54109770 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.538+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d541072c0 con 0x7f4d38038d00 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.539+0000 7f4d51600700 1 -- 172.21.15.2:0/3600612 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4d541072c0 con 0x7f4d38038d00 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.542+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 msgr2=0x7f4d3803b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.542+0000 7f4d5b200700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 0x7f4d3803b1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c006fd0 tx=0x7f4d3c006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 msgr2=0x7f4d5419d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000c00 tx=0x7f4d44011b40 comp rx=0 tx=0).stop 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 shutdown_connections 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f4d38038d00 0x7f4d3803b1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:01.223 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 --2- 172.21.15.2:0/3600612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d54109770 0x7f4d5419d540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 >> 172.21.15.2:0/3600612 conn(0x7f4d5407cba0 msgr2=0x7f4d54106bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 shutdown_connections 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:00.543+0000 7f4d5b200700 1 -- 172.21.15.2:0/3600612 wait complete. 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T18:22:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:Adding host smithi002... 2024-03-11T18:22:01.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:01 smithi002 ceph-mon[31328]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:02.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:02 smithi002 ceph-mon[31328]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:04.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:04 smithi002 ceph-mon[31328]: Deploying cephadm binary to smithi002 2024-03-11T18:22:09.343 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Added host 'smithi002' with addr '172.21.15.2' 2024-03-11T18:22:09.343 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.776+0000 7f5dfec00700 1 Processor -- start 2024-03-11T18:22:09.343 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.776+0000 7f5dfec00700 1 -- start start 2024-03-11T18:22:09.343 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.777+0000 7f5dfec00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:09.344 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.777+0000 7f5dfec00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df807b8f0 con 0x7f5df807ce70 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.777+0000 7f5dfd800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.777+0000 7f5dfd800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42230/0 (socket says 172.21.15.2:42230) 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.777+0000 7f5dfd800700 1 -- 172.21.15.2:0/2894938055 learned_addr learned my addr 172.21.15.2:0/2894938055 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.778+0000 7f5dfd800700 1 -- 172.21.15.2:0/2894938055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df807ba30 con 0x7f5df807ce70 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.778+0000 7f5dfd800700 1 --2- 172.21.15.2:0/2894938055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0098a0 tx=0x7f5dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2752ae6aad5b215 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5df7e00700 1 -- 172.21.15.2:0/2894938055 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dec004160 con 0x7f5df807ce70 2024-03-11T18:22:09.345 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5df7e00700 1 -- 172.21.15.2:0/2894938055 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dec0042c0 con 0x7f5df807ce70 2024-03-11T18:22:09.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5df7e00700 1 -- 172.21.15.2:0/2894938055 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dec011ac0 con 0x7f5df807ce70 2024-03-11T18:22:09.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5dfec00700 1 -- 172.21.15.2:0/2894938055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 msgr2=0x7f5df807b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:09.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5dfec00700 1 --2- 172.21.15.2:0/2894938055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0098a0 tx=0x7f5dec009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:09.346 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5dfec00700 1 -- 172.21.15.2:0/2894938055 shutdown_connections 2024-03-11T18:22:09.348 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5dfec00700 1 --2- 172.21.15.2:0/2894938055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df807b320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.779+0000 7f5dfec00700 1 -- 172.21.15.2:0/2894938055 >> 172.21.15.2:0/2894938055 conn(0x7f5df8102930 msgr2=0x7f5df8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.780+0000 7f5dfec00700 1 -- 172.21.15.2:0/2894938055 shutdown_connections 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.780+0000 7f5dfec00700 1 -- 172.21.15.2:0/2894938055 wait complete. 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.781+0000 7f5dfec00700 1 Processor -- start 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.781+0000 7f5dfec00700 1 -- start start 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.781+0000 7f5dfec00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.781+0000 7f5dfec00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81a1d00 con 0x7f5df807ce70 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.781+0000 7f5dfd800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5dfd800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42234/0 (socket says 172.21.15.2:42234) 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5dfd800700 1 -- 172.21.15.2:0/1536898117 learned_addr learned my addr 172.21.15.2:0/1536898117 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:09.349 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5dfd800700 1 -- 172.21.15.2:0/1536898117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec009550 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5dfd800700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009fb0 tx=0x7f5dec011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dec0038b0 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.782+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dec003a10 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.783+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df81a1f00 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.783+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dec019550 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.783+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df81a23a0 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.784+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df8109370 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.784+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f5dec01a430 con 0x7f5df807ce70 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.784+0000 7f5df6000700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 0x7f5de003b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:09.350 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.784+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5dec04cd90 con 0x7f5df807ce70 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.785+0000 7f5dfce00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 0x7f5de003b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.785+0000 7f5dfce00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 0x7f5de003b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006fd0 tx=0x7f5de4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:01.790+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5dec003b80 con 0x7f5df807ce70 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:02.019+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5df80620b0 con 0x7f5de0038cb0 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.621+0000 7f5df6000700 1 -- 172.21.15.2:0/1536898117 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5df80620b0 con 0x7f5de0038cb0 2024-03-11T18:22:09.352 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 msgr2=0x7f5de003b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 0x7f5de003b170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006fd0 tx=0x7f5de4006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 msgr2=0x7f5df81a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009fb0 tx=0x7f5dec011b40 comp rx=0 tx=0).stop 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 shutdown_connections 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5de0038cb0 0x7f5de003b170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 --2- 172.21.15.2:0/1536898117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5df807ce70 0x7f5df81a17c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 >> 172.21.15.2:0/1536898117 conn(0x7f5df8102930 msgr2=0x7f5df8104d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 shutdown_connections 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:08.625+0000 7f5dfec00700 1 -- 172.21.15.2:0/1536898117 wait complete. 2024-03-11T18:22:09.353 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mon service... 2024-03-11T18:22:09.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:09 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:09.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:09 smithi002 ceph-mon[31328]: Added host smithi002 2024-03-11T18:22:09.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:09 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:11.423 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T18:22:11.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.108+0000 7f5ebda00700 1 Processor -- start 2024-03-11T18:22:11.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.108+0000 7f5ebda00700 1 -- start start 2024-03-11T18:22:11.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.108+0000 7f5ebda00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:11.424 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.108+0000 7f5ebda00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb810a190 con 0x7f5eb81097a0 2024-03-11T18:22:11.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.109+0000 7f5eb6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:11.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.109+0000 7f5eb6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55856/0 (socket says 172.21.15.2:55856) 2024-03-11T18:22:11.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.109+0000 7f5eb6a00700 1 -- 172.21.15.2:0/3185643042 learned_addr learned my addr 172.21.15.2:0/3185643042 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:11.425 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.110+0000 7f5eb6a00700 1 -- 172.21.15.2:0/3185643042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb810a950 con 0x7f5eb81097a0 2024-03-11T18:22:11.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.110+0000 7f5eb6a00700 1 --2- 172.21.15.2:0/3185643042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098a0 tx=0x7f5ea8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b04a3252f02e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:11.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.110+0000 7f5eb5600700 1 -- 172.21.15.2:0/3185643042 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8004160 con 0x7f5eb81097a0 2024-03-11T18:22:11.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.110+0000 7f5eb5600700 1 -- 172.21.15.2:0/3185643042 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea80042c0 con 0x7f5eb81097a0 2024-03-11T18:22:11.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.110+0000 7f5eb5600700 1 -- 172.21.15.2:0/3185643042 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8011ac0 con 0x7f5eb81097a0 2024-03-11T18:22:11.426 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 -- 172.21.15.2:0/3185643042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 msgr2=0x7f5eb8109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3185643042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098a0 tx=0x7f5ea8009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 -- 172.21.15.2:0/3185643042 shutdown_connections 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3185643042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb8109bc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 -- 172.21.15.2:0/3185643042 >> 172.21.15.2:0/3185643042 conn(0x7f5eb807cd50 msgr2=0x7f5eb81074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.111+0000 7f5ebda00700 1 -- 172.21.15.2:0/3185643042 shutdown_connections 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.112+0000 7f5ebda00700 1 -- 172.21.15.2:0/3185643042 wait complete. 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.113+0000 7f5ebda00700 1 Processor -- start 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.113+0000 7f5ebda00700 1 -- start start 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.113+0000 7f5ebda00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.113+0000 7f5ebda00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb819da70 con 0x7f5eb81097a0 2024-03-11T18:22:11.431 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.114+0000 7f5eb6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.114+0000 7f5eb6a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55858/0 (socket says 172.21.15.2:55858) 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.114+0000 7f5eb6a00700 1 -- 172.21.15.2:0/3993328560 learned_addr learned my addr 172.21.15.2:0/3993328560 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.114+0000 7f5eb6a00700 1 -- 172.21.15.2:0/3993328560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8009550 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.114+0000 7f5eb6a00700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8011780 tx=0x7f5ea8011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.115+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8003af0 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.115+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ea8003c50 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.115+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ea80197e0 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.115+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb819dc70 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.115+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb819e110 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.116+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb810db80 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.116+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f5ea801aba0 con 0x7f5eb81097a0 2024-03-11T18:22:11.432 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.117+0000 7f5eaf400700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 0x7f5e9c03b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.117+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5ea804d0d0 con 0x7f5eb81097a0 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.117+0000 7f5eb6000700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 0x7f5e9c03b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.118+0000 7f5eb6000700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 0x7f5e9c03b1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4006fd0 tx=0x7f5ea4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.123+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ea8080d20 con 0x7f5eb81097a0 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.375+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5eb81a09f0 con 0x7f5e9c038d00 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.382+0000 7f5eaf400700 1 -- 172.21.15.2:0/3993328560 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5eb81a09f0 con 0x7f5e9c038d00 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 msgr2=0x7f5e9c03b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 0x7f5e9c03b1c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4006fd0 tx=0x7f5ea4006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 msgr2=0x7f5eb819d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8011780 tx=0x7f5ea8011b40 comp rx=0 tx=0).stop 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 shutdown_connections 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5e9c038d00 0x7f5e9c03b1c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 --2- 172.21.15.2:0/3993328560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb81097a0 0x7f5eb819d530 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:11.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 >> 172.21.15.2:0/3993328560 conn(0x7f5eb807cd50 msgr2=0x7f5eb8106bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:11.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 shutdown_connections 2024-03-11T18:22:11.434 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:10.386+0000 7f5ebda00700 1 -- 172.21.15.2:0/3993328560 wait complete. 2024-03-11T18:22:11.434 INFO:teuthology.orchestra.run.smithi002.stdout:Deploying unmanaged mgr service... 2024-03-11T18:22:11.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:11 smithi002 ceph-mon[31328]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:11.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:11 smithi002 ceph-mon[31328]: Saving service mon spec with placement count:5 2024-03-11T18:22:11.670 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:11 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:13.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:12 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:13.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:12 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:13.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:12 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:13.477 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.517+0000 7f69ea600700 1 Processor -- start 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.517+0000 7f69ea600700 1 -- start start 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.517+0000 7f69ea600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.517+0000 7f69ea600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e407b8b0 con 0x7f69e407ce30 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.518+0000 7f69e3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.518+0000 7f69e3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55866/0 (socket says 172.21.15.2:55866) 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.518+0000 7f69e3400700 1 -- 172.21.15.2:0/1787991626 learned_addr learned my addr 172.21.15.2:0/1787991626 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.518+0000 7f69e3400700 1 -- 172.21.15.2:0/1787991626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e407b9f0 con 0x7f69e407ce30 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.519+0000 7f69e3400700 1 --2- 172.21.15.2:0/1787991626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009e80 tx=0x7f69d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e72414a24721b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:13.478 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.519+0000 7f69e2000700 1 -- 172.21.15.2:0/1787991626 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d400e070 con 0x7f69e407ce30 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.519+0000 7f69e2000700 1 -- 172.21.15.2:0/1787991626 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69d400b8c0 con 0x7f69e407ce30 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.519+0000 7f69e2000700 1 -- 172.21.15.2:0/1787991626 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d40044a0 con 0x7f69e407ce30 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 -- 172.21.15.2:0/1787991626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 msgr2=0x7f69e407b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 --2- 172.21.15.2:0/1787991626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009e80 tx=0x7f69d400b330 comp rx=0 tx=0).stop 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 -- 172.21.15.2:0/1787991626 shutdown_connections 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 --2- 172.21.15.2:0/1787991626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e407b2e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 -- 172.21.15.2:0/1787991626 >> 172.21.15.2:0/1787991626 conn(0x7f69e4102b10 msgr2=0x7f69e4104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 -- 172.21.15.2:0/1787991626 shutdown_connections 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.520+0000 7f69ea600700 1 -- 172.21.15.2:0/1787991626 wait complete. 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.521+0000 7f69ea600700 1 Processor -- start 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69ea600700 1 -- start start 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69ea600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69ea600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e419d900 con 0x7f69e407ce30 2024-03-11T18:22:13.479 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69e3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69e3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55870/0 (socket says 172.21.15.2:55870) 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.522+0000 7f69e3400700 1 -- 172.21.15.2:0/1767125994 learned_addr learned my addr 172.21.15.2:0/1767125994 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69e3400700 1 -- 172.21.15.2:0/1767125994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4009b30 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69e3400700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009530 tx=0x7f69d400bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d400e040 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69d4003d60 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69d401c730 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e419db00 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.523+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e419e000 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.525+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f69d4012b30 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.525+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e410d990 con 0x7f69e407ce30 2024-03-11T18:22:13.480 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.525+0000 7f69dbe00700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 0x7f69c803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.525+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f69d404cc20 con 0x7f69e407ce30 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.525+0000 7f69e2a00700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 0x7f69c803b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.526+0000 7f69e2a00700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 0x7f69c803b1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.535+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69d4024c00 con 0x7f69e407ce30 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.777+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f69e4103c20 con 0x7f69c8038d00 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.786+0000 7f69dbe00700 1 -- 172.21.15.2:0/1767125994 <== mgr.14120 v2:172.21.15.2:6800/2981412802 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f69e4103c20 con 0x7f69c8038d00 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 msgr2=0x7f69c803b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 0x7f69c803b1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f69d0006fd0 tx=0x7f69d0006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 msgr2=0x7f69e419d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009530 tx=0x7f69d400bd10 comp rx=0 tx=0).stop 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 shutdown_connections 2024-03-11T18:22:13.481 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f69c8038d00 0x7f69c803b1c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:13.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 --2- 172.21.15.2:0/1767125994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e407ce30 0x7f69e419d3c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:13.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 >> 172.21.15.2:0/1767125994 conn(0x7f69e4102b10 msgr2=0x7f69e4103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:13.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 shutdown_connections 2024-03-11T18:22:13.482 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:12.789+0000 7f69ea600700 1 -- 172.21.15.2:0/1767125994 wait complete. 2024-03-11T18:22:14.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:14 smithi002 ceph-mon[31328]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:14.442 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:14 smithi002 ceph-mon[31328]: Saving service mgr spec with placement count:2 2024-03-11T18:22:14.442 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:14 smithi002 ceph-mon[31328]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:15.585 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.185+0000 7fa176e00700 1 Processor -- start 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.185+0000 7fa176e00700 1 -- start start 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.186+0000 7fa176e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.186+0000 7fa176e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17007ae50 con 0x7fa170108020 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.186+0000 7fa16fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.186+0000 7fa16fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55872/0 (socket says 172.21.15.2:55872) 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.186+0000 7fa16fe00700 1 -- 172.21.15.2:0/3884644954 learned_addr learned my addr 172.21.15.2:0/3884644954 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.187+0000 7fa16fe00700 1 -- 172.21.15.2:0/3884644954 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17010a950 con 0x7fa170108020 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.187+0000 7fa16fe00700 1 --2- 172.21.15.2:0/3884644954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098a0 tx=0x7fa160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2bb2374ad75de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.188+0000 7fa16ea00700 1 -- 172.21.15.2:0/3884644954 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160004160 con 0x7fa170108020 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.188+0000 7fa16ea00700 1 -- 172.21.15.2:0/3884644954 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1600042c0 con 0x7fa170108020 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.188+0000 7fa16ea00700 1 -- 172.21.15.2:0/3884644954 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160011a80 con 0x7fa170108020 2024-03-11T18:22:15.586 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 -- 172.21.15.2:0/3884644954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 msgr2=0x7fa17010a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 --2- 172.21.15.2:0/3884644954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1600098a0 tx=0x7fa160009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 -- 172.21.15.2:0/3884644954 shutdown_connections 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 --2- 172.21.15.2:0/3884644954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17010a410 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 -- 172.21.15.2:0/3884644954 >> 172.21.15.2:0/3884644954 conn(0x7fa170101a50 msgr2=0x7fa170103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 -- 172.21.15.2:0/3884644954 shutdown_connections 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.189+0000 7fa176e00700 1 -- 172.21.15.2:0/3884644954 wait complete. 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.190+0000 7fa176e00700 1 Processor -- start 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.190+0000 7fa176e00700 1 -- start start 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa176e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa176e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019da10 con 0x7fa170108020 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa16fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa16fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55874/0 (socket says 172.21.15.2:55874) 2024-03-11T18:22:15.587 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa16fe00700 1 -- 172.21.15.2:0/28381509 learned_addr learned my addr 172.21.15.2:0/28381509 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.191+0000 7fa16fe00700 1 -- 172.21.15.2:0/28381509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160009550 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa16fe00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa160009130 tx=0x7fa160011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa160003aa0 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa160003c00 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1600248a0 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17019dc10 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.192+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17019e0b0 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.194+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17010b790 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.194+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7fa16001ac10 con 0x7fa170108020 2024-03-11T18:22:15.588 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.194+0000 7fa16cc00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 0x7fa15403b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.194+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa16004cee0 con 0x7fa170108020 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.194+0000 7fa16f400700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 0x7fa15403b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.195+0000 7fa16f400700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 0x7fa15403b1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa15c006fd0 tx=0x7fa15c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.199+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa160018760 con 0x7fa170108020 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.440+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa17010b9a0 con 0x7fa170108020 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.446+0000 7fa16cc00700 1 -- 172.21.15.2:0/28381509 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa16000b3e0 con 0x7fa170108020 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 msgr2=0x7fa15403b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:15.591 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 0x7fa15403b1c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa15c006fd0 tx=0x7fa15c006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 msgr2=0x7fa17019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa160009130 tx=0x7fa160011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 shutdown_connections 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fa154038d00 0x7fa15403b1c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 --2- 172.21.15.2:0/28381509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa170108020 0x7fa17019d4d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.449+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 >> 172.21.15.2:0/28381509 conn(0x7fa170101a50 msgr2=0x7fa1701024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.450+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 shutdown_connections 2024-03-11T18:22:15.592 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:14.450+0000 7fa176e00700 1 -- 172.21.15.2:0/28381509 wait complete. 2024-03-11T18:22:15.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:15 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/28381509' entity='client.admin' 2024-03-11T18:22:15.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:15 smithi002 ceph-mon[31328]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:16.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:16 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:16.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:16 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3745276837' entity='client.admin' 2024-03-11T18:22:17.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.269+0000 7f5690a00700 1 Processor -- start 2024-03-11T18:22:17.212 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.270+0000 7f5690a00700 1 -- start start 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.270+0000 7f5690a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.270+0000 7f5690a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c07b8b0 con 0x7f568c07ce30 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.271+0000 7f5689e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.271+0000 7f5689e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55876/0 (socket says 172.21.15.2:55876) 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.271+0000 7f5689e00700 1 -- 172.21.15.2:0/1241327085 learned_addr learned my addr 172.21.15.2:0/1241327085 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.271+0000 7f5689e00700 1 -- 172.21.15.2:0/1241327085 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c07b9f0 con 0x7f568c07ce30 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.271+0000 7f5689e00700 1 --2- 172.21.15.2:0/1241327085 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f567c0098a0 tx=0x7f567c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498aab01b79884f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.272+0000 7f5689400700 1 -- 172.21.15.2:0/1241327085 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567c004160 con 0x7f568c07ce30 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.272+0000 7f5689400700 1 -- 172.21.15.2:0/1241327085 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f567c0042c0 con 0x7f568c07ce30 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.272+0000 7f5689400700 1 -- 172.21.15.2:0/1241327085 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567c011a80 con 0x7f568c07ce30 2024-03-11T18:22:17.213 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 -- 172.21.15.2:0/1241327085 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 msgr2=0x7f568c07b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 --2- 172.21.15.2:0/1241327085 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f567c0098a0 tx=0x7f567c009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 -- 172.21.15.2:0/1241327085 shutdown_connections 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 --2- 172.21.15.2:0/1241327085 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c07b2e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 -- 172.21.15.2:0/1241327085 >> 172.21.15.2:0/1241327085 conn(0x7f568c102b10 msgr2=0x7f568c104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 -- 172.21.15.2:0/1241327085 shutdown_connections 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.273+0000 7f5690a00700 1 -- 172.21.15.2:0/1241327085 wait complete. 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.274+0000 7f5690a00700 1 Processor -- start 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.274+0000 7f5690a00700 1 -- start start 2024-03-11T18:22:17.214 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5690a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5690a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c19d960 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5689e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5689e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55890/0 (socket says 172.21.15.2:55890) 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5689e00700 1 -- 172.21.15.2:0/3745276837 learned_addr learned my addr 172.21.15.2:0/3745276837 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.275+0000 7f5689e00700 1 -- 172.21.15.2:0/3745276837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c009550 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5689e00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f567c011740 tx=0x7f567c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567c003b20 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f567c01a910 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567c0248c0 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c19db60 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.276+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c19e000 con 0x7f568c07ce30 2024-03-11T18:22:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.277+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c10d990 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.277+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f567c01aa80 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.278+0000 7f5682a00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 0x7f567003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.278+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f567c04d0f0 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.278+0000 7f5683e00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 0x7f567003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.279+0000 7f5683e00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 0x7f567003b1c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5674006fd0 tx=0x7f5674006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.284+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f567c01adf0 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.515+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f568c10dba0 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.522+0000 7f5682a00700 1 -- 172.21.15.2:0/3745276837 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f567c01adf0 con 0x7f568c07ce30 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 msgr2=0x7f567003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 0x7f567003b1c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5674006fd0 tx=0x7f5674006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 msgr2=0x7f568c19d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f567c011740 tx=0x7f567c011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:17.216 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 shutdown_connections 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f5670038d00 0x7f567003b1c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 --2- 172.21.15.2:0/3745276837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f568c07ce30 0x7f568c19d420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.526+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 >> 172.21.15.2:0/3745276837 conn(0x7f568c102b10 msgr2=0x7f568c103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.527+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 shutdown_connections 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:16.527+0000 7f5690a00700 1 -- 172.21.15.2:0/3745276837 wait complete. 2024-03-11T18:22:17.217 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling the dashboard module... 2024-03-11T18:22:17.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:17 smithi002 ceph-mon[31328]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:19.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setuser ceph since I am not root 2024-03-11T18:22:19.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setgroup ceph since I am not root 2024-03-11T18:22:19.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:18 smithi002 ceph-mon[31328]: from='mgr.14120 172.21.15.2:0/4197962839' entity='mgr.x' 2024-03-11T18:22:19.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/254775445' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T18:22:19.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.792+0000 7f8969c00700 1 Processor -- start 2024-03-11T18:22:19.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.792+0000 7f8969c00700 1 -- start start 2024-03-11T18:22:19.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.793+0000 7f8969c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:19.671 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.793+0000 7f8969c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896407ae90 con 0x7f8964105dd0 2024-03-11T18:22:19.673 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.793+0000 7f8962a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:19.673 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.793+0000 7f8962a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58208/0 (socket says 172.21.15.2:58208) 2024-03-11T18:22:19.673 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.793+0000 7f8962a00700 1 -- 172.21.15.2:0/1903123777 learned_addr learned my addr 172.21.15.2:0/1903123777 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:19.673 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.794+0000 7f8962a00700 1 -- 172.21.15.2:0/1903123777 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8964108740 con 0x7f8964105dd0 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.794+0000 7f8962a00700 1 --2- 172.21.15.2:0/1903123777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89540098a0 tx=0x7f8954009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51345cdac4a84233 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.794+0000 7f8961600700 1 -- 172.21.15.2:0/1903123777 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8954004160 con 0x7f8964105dd0 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.794+0000 7f8961600700 1 -- 172.21.15.2:0/1903123777 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89540042c0 con 0x7f8964105dd0 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.794+0000 7f8961600700 1 -- 172.21.15.2:0/1903123777 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8954011a80 con 0x7f8964105dd0 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 -- 172.21.15.2:0/1903123777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 msgr2=0x7f8964108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 --2- 172.21.15.2:0/1903123777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89540098a0 tx=0x7f8954009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 -- 172.21.15.2:0/1903123777 shutdown_connections 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 --2- 172.21.15.2:0/1903123777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f8964108200 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 -- 172.21.15.2:0/1903123777 >> 172.21.15.2:0/1903123777 conn(0x7f8964101a50 msgr2=0x7f8964103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 -- 172.21.15.2:0/1903123777 shutdown_connections 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.795+0000 7f8969c00700 1 -- 172.21.15.2:0/1903123777 wait complete. 2024-03-11T18:22:19.674 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.797+0000 7f8969c00700 1 Processor -- start 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.797+0000 7f8969c00700 1 -- start start 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.797+0000 7f8969c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.797+0000 7f8969c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89641a1de0 con 0x7f8964105dd0 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.797+0000 7f8962a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.798+0000 7f8962a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58224/0 (socket says 172.21.15.2:58224) 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.798+0000 7f8962a00700 1 -- 172.21.15.2:0/254775445 learned_addr learned my addr 172.21.15.2:0/254775445 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.798+0000 7f8962a00700 1 -- 172.21.15.2:0/254775445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954009550 con 0x7f8964105dd0 2024-03-11T18:22:19.678 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.798+0000 7f8962a00700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8954009130 tx=0x7f8954011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.798+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8954003aa0 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.799+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8954003c00 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.799+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89540248a0 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.799+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89641a1fe0 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.799+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89641a2480 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.800+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 45769+0+0 (secure 0 0 0) 0x7f895401ac10 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.800+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896419b0b0 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.800+0000 7f895b400700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 0x7f894803b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.800+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f895404cf00 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.801+0000 7f8962000700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 0x7f894803b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.801+0000 7f8962000700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 0x7f894803b1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8950006fd0 tx=0x7f8950006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:17.806+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8954024a60 con 0x7f8964105dd0 2024-03-11T18:22:19.679 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.096+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8964109650 con 0x7f8964105dd0 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.832+0000 7f895b400700 1 -- 172.21.15.2:0/254775445 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f895402c350 con 0x7f8964105dd0 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.837+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 msgr2=0x7f894803b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.837+0000 7f8969c00700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 0x7f894803b1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8950006fd0 tx=0x7f8950006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.837+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 msgr2=0x7f89641a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.837+0000 7f8969c00700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8954009130 tx=0x7f8954011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.838+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 shutdown_connections 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.838+0000 7f8969c00700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f8948038d00 0x7f894803b1c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.838+0000 7f8969c00700 1 --2- 172.21.15.2:0/254775445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8964105dd0 0x7f89641a18a0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8954009130 tx=0x7f8954011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.838+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 >> 172.21.15.2:0/254775445 conn(0x7f8964101a50 msgr2=0x7f8964103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.839+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 shutdown_connections 2024-03-11T18:22:19.680 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:18.839+0000 7f8969c00700 1 -- 172.21.15.2:0/254775445 wait complete. 2024-03-11T18:22:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/254775445' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T18:22:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:19 smithi002 ceph-mon[31328]: mgrmap e9: x(active, since 25s) 2024-03-11T18:22:20.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:19.661+0000 7f232e15b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:22:20.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:19.793+0000 7f232e15b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:22:20.870 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:20.624+0000 7f232e15b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:22:20.870 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:20.753+0000 7f232e15b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:22:21.200 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:20.942+0000 7f232e15b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:22:21.200 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.076+0000 7f232e15b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:22:21.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2500197600' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T18:22:21.502 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.198+0000 7f232e15b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:22:21.825 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.560+0000 7f232e15b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:22:21.825 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.706+0000 7f232e15b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:22:21.825 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.823+0000 7f232e15b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:22:21.986 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:22:21.987 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.026+0000 7fd117800700 1 Processor -- start 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.027+0000 7fd117800700 1 -- start start 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.027+0000 7fd117800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.027+0000 7fd117800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11007ae90 con 0x7fd110107fe0 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.028+0000 7fd114e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.028+0000 7fd114e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58256/0 (socket says 172.21.15.2:58256) 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.028+0000 7fd114e00700 1 -- 172.21.15.2:0/3007468189 learned_addr learned my addr 172.21.15.2:0/3007468189 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.028+0000 7fd114e00700 1 -- 172.21.15.2:0/3007468189 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11010a910 con 0x7fd110107fe0 2024-03-11T18:22:21.989 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.028+0000 7fd114e00700 1 --2- 172.21.15.2:0/3007468189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1040098a0 tx=0x7fd104009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02ef72a96118768 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:21.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.029+0000 7fd10f400700 1 -- 172.21.15.2:0/3007468189 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd104004160 con 0x7fd110107fe0 2024-03-11T18:22:21.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.029+0000 7fd10f400700 1 -- 172.21.15.2:0/3007468189 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1040042c0 con 0x7fd110107fe0 2024-03-11T18:22:21.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.029+0000 7fd10f400700 1 -- 172.21.15.2:0/3007468189 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd104011a80 con 0x7fd110107fe0 2024-03-11T18:22:21.990 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.029+0000 7fd117800700 1 -- 172.21.15.2:0/3007468189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 msgr2=0x7fd11010a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:21.993 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.029+0000 7fd117800700 1 --2- 172.21.15.2:0/3007468189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd1040098a0 tx=0x7fd104009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.030+0000 7fd117800700 1 -- 172.21.15.2:0/3007468189 shutdown_connections 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.030+0000 7fd117800700 1 --2- 172.21.15.2:0/3007468189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11010a3d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.030+0000 7fd117800700 1 -- 172.21.15.2:0/3007468189 >> 172.21.15.2:0/3007468189 conn(0x7fd110101a50 msgr2=0x7fd110103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.030+0000 7fd117800700 1 -- 172.21.15.2:0/3007468189 shutdown_connections 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.030+0000 7fd117800700 1 -- 172.21.15.2:0/3007468189 wait complete. 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.031+0000 7fd117800700 1 Processor -- start 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd117800700 1 -- start start 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd117800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd117800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019da50 con 0x7fd110107fe0 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd114e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:21.994 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd114e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58270/0 (socket says 172.21.15.2:58270) 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.032+0000 7fd114e00700 1 -- 172.21.15.2:0/2500197600 learned_addr learned my addr 172.21.15.2:0/2500197600 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd114e00700 1 -- 172.21.15.2:0/2500197600 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd104009550 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd114e00700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd104003f20 tx=0x7fd104011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1040039f0 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd104003b50 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd10401a6d0 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.033+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11019dc50 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.034+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11019e0f0 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.035+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11010b7b0 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.035+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fd104024430 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.035+0000 7fd10d600700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 0x7fd0f803b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.036+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd10404d070 con 0x7fd110107fe0 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.036+0000 7fd10fe00700 1 -- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 msgr2=0x7fd0f803b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:21.995 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.036+0000 7fd10fe00700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 0x7fd0f803b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.041+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd104018a40 con 0x7fd110107fe0 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.237+0000 7fd10fe00700 1 -- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 msgr2=0x7fd0f803b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.237+0000 7fd10fe00700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 0x7fd0f803b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.323+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd1101a0d50 con 0x7fd110107fe0 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.324+0000 7fd10d600700 1 -- 172.21.15.2:0/2500197600 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd104018350 con 0x7fd110107fe0 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 msgr2=0x7fd0f803b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:22:21.996 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 0x7fd0f803b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:21.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 msgr2=0x7fd11019d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:21.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd104003f20 tx=0x7fd104011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:21.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 shutdown_connections 2024-03-11T18:22:21.997 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7fd0f8038d10 0x7fd0f803b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 --2- 172.21.15.2:0/2500197600 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd110107fe0 0x7fd11019d510 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.326+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 >> 172.21.15.2:0/2500197600 conn(0x7fd110101a50 msgr2=0x7fd110102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.327+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 shutdown_connections 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:21.327+0000 7fd117800700 1 -- 172.21.15.2:0/2500197600 wait complete. 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for the mgr to restart... 2024-03-11T18:22:21.998 INFO:teuthology.orchestra.run.smithi002.stdout:Waiting for mgr epoch 9... 2024-03-11T18:22:22.148 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:21.952+0000 7f232e15b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:22:22.425 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:22.184+0000 7f232e15b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:22:23.995 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:23.513+0000 7f232e15b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:22:23.995 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:23.690+0000 7f232e15b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:22:24.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:24.118+0000 7f232e15b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:22:24.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:24.242+0000 7f232e15b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:22:24.764 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:24.509+0000 7f232e15b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:22:25.154 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:24.762+0000 7f232e15b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:22:25.154 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:25.152+0000 7f232e15b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:22:25.751 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:25.487+0000 7f232e15b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:22:26.251 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:25.814+0000 7f232e15b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:22:27.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:26.895+0000 7f232e15b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:22:28.001 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:27.617+0000 7f232e15b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:22:28.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:27.753+0000 7f232e15b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:22:28.501 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:22:28.046+0000 7f232e15b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:22:28.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: Active manager daemon x restarted 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: Activating manager daemon x 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: mgrmap e10: x(active, starting, since 0.00875661s) 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:22:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:28 smithi002 ceph-mon[31328]: Manager daemon x is now available 2024-03-11T18:22:29.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:29.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:29.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:22:29.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:22:29.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:29 smithi002 ceph-mon[31328]: mgrmap e11: x(active, since 1.01007s) 2024-03-11T18:22:29.722 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout { 2024-03-11T18:22:29.731 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T18:22:29.731 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T18:22:29.731 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout } 2024-03-11T18:22:29.731 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.581+0000 7f3a05e00700 1 Processor -- start 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.581+0000 7f3a05e00700 1 -- start start 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.582+0000 7f3a05e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.582+0000 7f3a05e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0007b7e0 con 0x7f3a0007cdf0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.582+0000 7f39fea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.582+0000 7f39fea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58286/0 (socket says 172.21.15.2:58286) 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.582+0000 7f39fea00700 1 -- 172.21.15.2:0/620555645 learned_addr learned my addr 172.21.15.2:0/620555645 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.583+0000 7f39fea00700 1 -- 172.21.15.2:0/620555645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0007b920 con 0x7f3a0007cdf0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.583+0000 7f39fea00700 1 --2- 172.21.15.2:0/620555645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009e80 tx=0x7f39f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb358faba603ab01 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.583+0000 7f39fd600700 1 -- 172.21.15.2:0/620555645 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39f000e070 con 0x7f3a0007cdf0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.583+0000 7f39fd600700 1 -- 172.21.15.2:0/620555645 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39f000b8c0 con 0x7f3a0007cdf0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.583+0000 7f39fd600700 1 -- 172.21.15.2:0/620555645 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39f00044a0 con 0x7f3a0007cdf0 2024-03-11T18:22:29.732 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.584+0000 7f3a05e00700 1 -- 172.21.15.2:0/620555645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 msgr2=0x7f3a0007b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.584+0000 7f3a05e00700 1 --2- 172.21.15.2:0/620555645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009e80 tx=0x7f39f000b330 comp rx=0 tx=0).stop 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.584+0000 7f3a05e00700 1 -- 172.21.15.2:0/620555645 shutdown_connections 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.584+0000 7f3a05e00700 1 --2- 172.21.15.2:0/620555645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0007b2a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.584+0000 7f3a05e00700 1 -- 172.21.15.2:0/620555645 >> 172.21.15.2:0/620555645 conn(0x7f3a00102b10 msgr2=0x7f3a00104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.585+0000 7f3a05e00700 1 -- 172.21.15.2:0/620555645 shutdown_connections 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.585+0000 7f3a05e00700 1 -- 172.21.15.2:0/620555645 wait complete. 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.586+0000 7f3a05e00700 1 Processor -- start 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.586+0000 7f3a05e00700 1 -- start start 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.586+0000 7f3a05e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.586+0000 7f3a05e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0019d910 con 0x7f3a0007cdf0 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.587+0000 7f39fea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.587+0000 7f39fea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58302/0 (socket says 172.21.15.2:58302) 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.587+0000 7f39fea00700 1 -- 172.21.15.2:0/1125535780 learned_addr learned my addr 172.21.15.2:0/1125535780 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:29.733 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.587+0000 7f39fea00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f0009b30 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.587+0000 7f39fea00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009530 tx=0x7f39f000bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.588+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39f000e040 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.588+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39f0003d60 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.588+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0019db10 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.588+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39f001c730 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.588+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0019e010 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.589+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f39f0012430 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.589+0000 7f3a04a00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.589+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39e403b930 con 0x7f39e4038d60 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.590+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f39f004cc90 con 0x7f3a0007cdf0 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.590+0000 7f39fe000700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:29.734 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.590+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.791+0000 7f39fe000700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:22.791+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:23.192+0000 7f39fe000700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:23.192+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:23.993+0000 7f39fe000700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:23.993+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:25.595+0000 7f39fe000700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/2981412802 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:25.595+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:28.055+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mgrmap(e 10) v1 ==== 45549+0+0 (secure 0 0 0) 0x7f39f0012b30 con 0x7f3a0007cdf0 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:28.056+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 msgr2=0x7f39e403b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:28.056+0000 7f3a04a00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.056+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 45676+0+0 (secure 0 0 0) 0x7f39f004e210 con 0x7f3a0007cdf0 2024-03-11T18:22:29.735 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.056+0000 7f3a04a00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 0x7f39e403ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.056+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39e403b930 con 0x7f39e403c810 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.057+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 0x7f39e403ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.059+0000 7f39fe000700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 0x7f39e403ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39ec003a10 tx=0x7f39ec009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.060+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f39e403b930 con 0x7f39e403c810 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.072+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3a0010d970 con 0x7f39e403c810 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a04a00700 1 -- 172.21.15.2:0/1125535780 <== mgr.14152 v2:172.21.15.2:6800/1446641624 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f3a0010d970 con 0x7f39e403c810 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 msgr2=0x7f39e403ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 0x7f39e403ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f39ec003a10 tx=0x7f39ec009270 comp rx=0 tx=0).stop 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 msgr2=0x7f3a0019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009530 tx=0x7f39f000bd10 comp rx=0 tx=0).stop 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 shutdown_connections 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f39e403c810 0x7f39e403ec00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a0007cdf0 0x7f3a0019d3d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:29.736 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 --2- 172.21.15.2:0/1125535780 >> [v2:172.21.15.2:6800/2981412802,v1:172.21.15.2:6801/2981412802] conn(0x7f39e4038d60 0x7f39e403b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:29.737 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.073+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 >> 172.21.15.2:0/1125535780 conn(0x7f3a00102b10 msgr2=0x7f3a001034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:29.737 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.074+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 shutdown_connections 2024-03-11T18:22:29.737 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:29.074+0000 7f3a05e00700 1 -- 172.21.15.2:0/1125535780 wait complete. 2024-03-11T18:22:29.737 INFO:teuthology.orchestra.run.smithi002.stdout:mgr epoch 9 is available 2024-03-11T18:22:29.737 INFO:teuthology.orchestra.run.smithi002.stdout:Generating a dashboard self-signed certificate... 2024-03-11T18:22:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:30 smithi002 ceph-mon[31328]: [11/Mar/2024:18:22:29] ENGINE Bus STARTING 2024-03-11T18:22:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:30 smithi002 ceph-mon[31328]: [11/Mar/2024:18:22:29] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T18:22:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:30 smithi002 ceph-mon[31328]: [11/Mar/2024:18:22:29] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T18:22:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:30 smithi002 ceph-mon[31328]: [11/Mar/2024:18:22:29] ENGINE Bus STARTED 2024-03-11T18:22:31.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:31 smithi002 ceph-mon[31328]: mgrmap e12: x(active, since 2s) 2024-03-11T18:22:31.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:31 smithi002 ceph-mon[31328]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:31.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:31 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:31.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:31 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:31.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T18:22:31.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.479+0000 7f434ce00700 1 Processor -- start 2024-03-11T18:22:31.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.479+0000 7f434ce00700 1 -- start start 2024-03-11T18:22:31.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.480+0000 7f434ce00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:31.549 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.480+0000 7f434ce00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434810e9f0 con 0x7f434810e000 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.480+0000 7f4345e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.480+0000 7f4345e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47738/0 (socket says 172.21.15.2:47738) 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.480+0000 7f4345e00700 1 -- 172.21.15.2:0/2005068350 learned_addr learned my addr 172.21.15.2:0/2005068350 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.481+0000 7f4345e00700 1 -- 172.21.15.2:0/2005068350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434810eb30 con 0x7f434810e000 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.481+0000 7f4345e00700 1 --2- 172.21.15.2:0/2005068350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43380098a0 tx=0x7f4338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f577cb1741d6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.481+0000 7f4344a00700 1 -- 172.21.15.2:0/2005068350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4338004160 con 0x7f434810e000 2024-03-11T18:22:31.552 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.481+0000 7f4344a00700 1 -- 172.21.15.2:0/2005068350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43380042c0 con 0x7f434810e000 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f4344a00700 1 -- 172.21.15.2:0/2005068350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4338011a80 con 0x7f434810e000 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 -- 172.21.15.2:0/2005068350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 msgr2=0x7f434810e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 --2- 172.21.15.2:0/2005068350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f43380098a0 tx=0x7f4338009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 -- 172.21.15.2:0/2005068350 shutdown_connections 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 --2- 172.21.15.2:0/2005068350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f434810e420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 -- 172.21.15.2:0/2005068350 >> 172.21.15.2:0/2005068350 conn(0x7f434807cdf0 msgr2=0x7f434807f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 -- 172.21.15.2:0/2005068350 shutdown_connections 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.482+0000 7f434ce00700 1 -- 172.21.15.2:0/2005068350 wait complete. 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f434ce00700 1 Processor -- start 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f434ce00700 1 -- start start 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f434ce00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f434ce00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43481a1e30 con 0x7f434810e000 2024-03-11T18:22:31.553 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f4345e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:31.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f4345e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47750/0 (socket says 172.21.15.2:47750) 2024-03-11T18:22:31.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.484+0000 7f4345e00700 1 -- 172.21.15.2:0/720644831 learned_addr learned my addr 172.21.15.2:0/720644831 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:31.554 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f4345e00700 1 -- 172.21.15.2:0/720644831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338009550 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f4345e00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4338000c00 tx=0x7f4338011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4338003aa0 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4338003c00 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4338024930 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.485+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43481a2030 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.486+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43481a24d0 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.487+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f433801aca0 con 0x7f434810e000 2024-03-11T18:22:31.556 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.487+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4348111f90 con 0x7f434810e000 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.487+0000 7f433ea00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 0x7f432c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.487+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f433804cf10 con 0x7f434810e000 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.487+0000 7f4345400700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 0x7f432c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.488+0000 7f4345400700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 0x7f432c03b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.493+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4338018710 con 0x7f434810e000 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.727+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f43481a4db0 con 0x7f432c038d60 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.864+0000 7f433ea00700 1 -- 172.21.15.2:0/720644831 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f43481a4db0 con 0x7f432c038d60 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 msgr2=0x7f432c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 0x7f432c03b220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 msgr2=0x7f43481a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4338000c00 tx=0x7f4338011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 shutdown_connections 2024-03-11T18:22:31.557 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.867+0000 7f434ce00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f432c038d60 0x7f432c03b220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:31.558 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.868+0000 7f434ce00700 1 --2- 172.21.15.2:0/720644831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f434810e000 0x7f43481a18f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:31.558 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.868+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 >> 172.21.15.2:0/720644831 conn(0x7f434807cdf0 msgr2=0x7f434807e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:31.558 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.868+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 shutdown_connections 2024-03-11T18:22:31.558 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:30.868+0000 7f434ce00700 1 -- 172.21.15.2:0/720644831 wait complete. 2024-03-11T18:22:31.558 INFO:teuthology.orchestra.run.smithi002.stdout:Creating initial admin user... 2024-03-11T18:22:32.460 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:32 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$c9IwPrdNclU.iBs0Fheqxe6j9z1PfqpTI21trgRCzULdmCEntYfqC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710181352, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.173+0000 7fc33b800700 1 Processor -- start 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.173+0000 7fc33b800700 1 -- start start 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.173+0000 7fc33b800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.174+0000 7fc33b800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334107f60 con 0x7fc334107570 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.174+0000 7fc338e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:33.446 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.174+0000 7fc338e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47762/0 (socket says 172.21.15.2:47762) 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.174+0000 7fc338e00700 1 -- 172.21.15.2:0/2420527743 learned_addr learned my addr 172.21.15.2:0/2420527743 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.174+0000 7fc338e00700 1 -- 172.21.15.2:0/2420527743 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334108720 con 0x7fc334107570 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.175+0000 7fc338e00700 1 --2- 172.21.15.2:0/2420527743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc3280098a0 tx=0x7fc328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f40bb54e8baa60 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.175+0000 7fc333400700 1 -- 172.21.15.2:0/2420527743 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc328004160 con 0x7fc334107570 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.175+0000 7fc333400700 1 -- 172.21.15.2:0/2420527743 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3280042c0 con 0x7fc334107570 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.175+0000 7fc333400700 1 -- 172.21.15.2:0/2420527743 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc328011a80 con 0x7fc334107570 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 -- 172.21.15.2:0/2420527743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 msgr2=0x7fc334107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 --2- 172.21.15.2:0/2420527743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc3280098a0 tx=0x7fc328009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 -- 172.21.15.2:0/2420527743 shutdown_connections 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 --2- 172.21.15.2:0/2420527743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc334107990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 -- 172.21.15.2:0/2420527743 >> 172.21.15.2:0/2420527743 conn(0x7fc334102b10 msgr2=0x7fc334104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 -- 172.21.15.2:0/2420527743 shutdown_connections 2024-03-11T18:22:33.447 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.176+0000 7fc33b800700 1 -- 172.21.15.2:0/2420527743 wait complete. 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.178+0000 7fc33b800700 1 Processor -- start 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.178+0000 7fc33b800700 1 -- start start 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc33b800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc33b800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33419b790 con 0x7fc334107570 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc338e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc338e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47766/0 (socket says 172.21.15.2:47766) 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc338e00700 1 -- 172.21.15.2:0/3383314027 learned_addr learned my addr 172.21.15.2:0/3383314027 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.179+0000 7fc338e00700 1 -- 172.21.15.2:0/3383314027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328009550 con 0x7fc334107570 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc338e00700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc328011740 tx=0x7fc328011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3280039e0 con 0x7fc334107570 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc328003b40 con 0x7fc334107570 2024-03-11T18:22:33.448 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc328024740 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33419b990 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.180+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33419be30 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.181+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7fc32801ab30 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.181+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33410b950 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.182+0000 7fc331600700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 0x7fc31c03b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.182+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc32804d030 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.182+0000 7fc333e00700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 0x7fc31c03b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.183+0000 7fc333e00700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 0x7fc31c03b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc320006fd0 tx=0x7fc320006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.188+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc328080d20 con 0x7fc334107570 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.457+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc3340630c0 con 0x7fc31c038d60 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.729+0000 7fc331600700 1 -- 172.21.15.2:0/3383314027 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc3340630c0 con 0x7fc31c038d60 2024-03-11T18:22:33.449 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 msgr2=0x7fc31c03b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 0x7fc31c03b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc320006fd0 tx=0x7fc320006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 msgr2=0x7fc33419b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc328011740 tx=0x7fc328011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 shutdown_connections 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc31c038d60 0x7fc31c03b220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 --2- 172.21.15.2:0/3383314027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc334107570 0x7fc33419b250 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 >> 172.21.15.2:0/3383314027 conn(0x7fc334102b10 msgr2=0x7fc334103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 shutdown_connections 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:32.732+0000 7fc33b800700 1 -- 172.21.15.2:0/3383314027 wait complete. 2024-03-11T18:22:33.450 INFO:teuthology.orchestra.run.smithi002.stdout:Fetching dashboard port number... 2024-03-11T18:22:33.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:33 smithi002 ceph-mon[31328]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:33.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:33.989 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:34.892 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:34 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2366819665' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.037+0000 7f0986600700 1 Processor -- start 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.037+0000 7f0986600700 1 -- start start 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.038+0000 7f0986600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.038+0000 7f0986600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098007add0 con 0x7f09801080b0 2024-03-11T18:22:34.911 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.038+0000 7f097f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:34.912 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.038+0000 7f097f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47772/0 (socket says 172.21.15.2:47772) 2024-03-11T18:22:34.912 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.038+0000 7f097f400700 1 -- 172.21.15.2:0/2218942763 learned_addr learned my addr 172.21.15.2:0/2218942763 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.039+0000 7f097f400700 1 -- 172.21.15.2:0/2218942763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098007af10 con 0x7f09801080b0 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.039+0000 7f097f400700 1 --2- 172.21.15.2:0/2218942763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09700098a0 tx=0x7f0970009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1b0b5aaaa14a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.040+0000 7f097e000700 1 -- 172.21.15.2:0/2218942763 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0970004160 con 0x7f09801080b0 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.040+0000 7f097e000700 1 -- 172.21.15.2:0/2218942763 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09700042c0 con 0x7f09801080b0 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.040+0000 7f097e000700 1 -- 172.21.15.2:0/2218942763 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0970011a80 con 0x7f09801080b0 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 -- 172.21.15.2:0/2218942763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 msgr2=0x7f098010a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 --2- 172.21.15.2:0/2218942763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09700098a0 tx=0x7f0970009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 -- 172.21.15.2:0/2218942763 shutdown_connections 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 --2- 172.21.15.2:0/2218942763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098010a4a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 -- 172.21.15.2:0/2218942763 >> 172.21.15.2:0/2218942763 conn(0x7f0980101aa0 msgr2=0x7f0980103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 -- 172.21.15.2:0/2218942763 shutdown_connections 2024-03-11T18:22:34.917 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.041+0000 7f0986600700 1 -- 172.21.15.2:0/2218942763 wait complete. 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.043+0000 7f0986600700 1 Processor -- start 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.043+0000 7f0986600700 1 -- start start 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.043+0000 7f0986600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.043+0000 7f0986600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098019b840 con 0x7f09801080b0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.044+0000 7f097f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.044+0000 7f097f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47788/0 (socket says 172.21.15.2:47788) 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.044+0000 7f097f400700 1 -- 172.21.15.2:0/2366819665 learned_addr learned my addr 172.21.15.2:0/2366819665 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.044+0000 7f097f400700 1 -- 172.21.15.2:0/2366819665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009550 con 0x7f09801080b0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f097f400700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0970009130 tx=0x7f0970011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0970003aa0 con 0x7f09801080b0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0970003c00 con 0x7f09801080b0 2024-03-11T18:22:34.918 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098019ba40 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f09700248a0 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.045+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098019bee0 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.046+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098010b7b0 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.046+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 45782+0+0 (secure 0 0 0) 0x7f0970024b20 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.047+0000 7f0977e00700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 0x7f096403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.047+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f097004cf80 con 0x7f09801080b0 2024-03-11T18:22:34.919 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.047+0000 7f097ea00700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 0x7f096403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.048+0000 7f097ea00700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 0x7f096403b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f096c006fd0 tx=0x7f096c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.052+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0970024dd0 con 0x7f09801080b0 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.281+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f098019e7c0 con 0x7f09801080b0 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.282+0000 7f0977e00700 1 -- 172.21.15.2:0/2366819665 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0970018ba0 con 0x7f09801080b0 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 msgr2=0x7f096403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 0x7f096403b220 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f096c006fd0 tx=0x7f096c006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 msgr2=0x7f098019b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0970009130 tx=0x7f0970011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 shutdown_connections 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0964038d60 0x7f096403b220 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:34.920 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 --2- 172.21.15.2:0/2366819665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09801080b0 0x7f098019b300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:34.921 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 >> 172.21.15.2:0/2366819665 conn(0x7f0980101aa0 msgr2=0x7f0980102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:34.921 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 shutdown_connections 2024-03-11T18:22:34.921 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:34.286+0000 7f0986600700 1 -- 172.21.15.2:0/2366819665 wait complete. 2024-03-11T18:22:34.941 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T18:22:34.941 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout disabled 2024-03-11T18:22:34.961 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T18:22:34.961 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stdout inactive 2024-03-11T18:22:34.962 INFO:teuthology.orchestra.run.smithi002.stdout:firewalld.service is not enabled 2024-03-11T18:22:34.962 INFO:teuthology.orchestra.run.smithi002.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout:Ceph Dashboard is now available at: 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout: URL: https://smithi002.front.sepia.ceph.com:8443/ 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout: User: admin 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout: Password: 9ynr09dd7o 2024-03-11T18:22:34.965 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:34.966 INFO:teuthology.orchestra.run.smithi002.stdout:Saving cluster configuration to /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config directory 2024-03-11T18:22:34.966 INFO:teuthology.orchestra.run.smithi002.stdout:Enabling autotune for osd_memory_target 2024-03-11T18:22:36.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:35 smithi002 ceph-mon[31328]: mgrmap e13: x(active, since 6s) 2024-03-11T18:22:37.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.561+0000 7f5596a00700 1 Processor -- start 2024-03-11T18:22:37.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.561+0000 7f5596a00700 1 -- start start 2024-03-11T18:22:37.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.562+0000 7f5596a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:37.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.562+0000 7f5596a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590106e60 con 0x7f5590106470 2024-03-11T18:22:37.451 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.562+0000 7f558fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.562+0000 7f558fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47800/0 (socket says 172.21.15.2:47800) 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.562+0000 7f558fe00700 1 -- 172.21.15.2:0/2791856947 learned_addr learned my addr 172.21.15.2:0/2791856947 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.563+0000 7f558fe00700 1 -- 172.21.15.2:0/2791856947 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5590107670 con 0x7f5590106470 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.563+0000 7f558fe00700 1 --2- 172.21.15.2:0/2791856947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55800098a0 tx=0x7f5580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba33e3f7e0aa99e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.563+0000 7f558f400700 1 -- 172.21.15.2:0/2791856947 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5580004160 con 0x7f5590106470 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.563+0000 7f558f400700 1 -- 172.21.15.2:0/2791856947 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55800042c0 con 0x7f5590106470 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.563+0000 7f558f400700 1 -- 172.21.15.2:0/2791856947 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5580011a80 con 0x7f5590106470 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 -- 172.21.15.2:0/2791856947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 msgr2=0x7f5590106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 --2- 172.21.15.2:0/2791856947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f55800098a0 tx=0x7f5580009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 -- 172.21.15.2:0/2791856947 shutdown_connections 2024-03-11T18:22:37.452 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 --2- 172.21.15.2:0/2791856947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f5590106890 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 -- 172.21.15.2:0/2791856947 >> 172.21.15.2:0/2791856947 conn(0x7f5590101a50 msgr2=0x7f5590103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 -- 172.21.15.2:0/2791856947 shutdown_connections 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.564+0000 7f5596a00700 1 -- 172.21.15.2:0/2791856947 wait complete. 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.565+0000 7f5596a00700 1 Processor -- start 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f5596a00700 1 -- start start 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f5596a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f5596a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559019db40 con 0x7f5590106470 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f558fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f558fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47810/0 (socket says 172.21.15.2:47810) 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.566+0000 7f558fe00700 1 -- 172.21.15.2:0/1489410564 learned_addr learned my addr 172.21.15.2:0/1489410564 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f558fe00700 1 -- 172.21.15.2:0/1489410564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5580009550 con 0x7f5590106470 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f558fe00700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5580011740 tx=0x7f5580011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:37.453 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55800039e0 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5580003b40 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55800196c0 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559019dd40 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.567+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559019e1e0 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.569+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f558001a430 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.569+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5590197910 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.569+0000 7f558d600700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 0x7f557403b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.569+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f558004d1b0 con 0x7f5590106470 2024-03-11T18:22:37.454 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.570+0000 7f5587400700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 0x7f557403b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.570+0000 7f5587400700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 0x7f557403b260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5578006fd0 tx=0x7f5578006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.575+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f558002da40 con 0x7f5590106470 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.801+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f559010a870 con 0x7f5590106470 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.802+0000 7f558d600700 1 -- 172.21.15.2:0/1489410564 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f558002d350 con 0x7f5590106470 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.804+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 msgr2=0x7f557403b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.804+0000 7f5596a00700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 0x7f557403b260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5578006fd0 tx=0x7f5578006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.804+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 msgr2=0x7f559019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.804+0000 7f5596a00700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5580011740 tx=0x7f5580011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 shutdown_connections 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5574038da0 0x7f557403b260 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 --2- 172.21.15.2:0/1489410564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5590106470 0x7f559019d600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:37.455 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 >> 172.21.15.2:0/1489410564 conn(0x7f5590101a50 msgr2=0x7f55901028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:37.456 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 shutdown_connections 2024-03-11T18:22:37.456 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:36.805+0000 7f5596a00700 1 -- 172.21.15.2:0/1489410564 wait complete. 2024-03-11T18:22:39.108 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.052+0000 7f65cb000700 1 Processor -- start 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.052+0000 7f65cb000700 1 -- start start 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.053+0000 7f65cb000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.053+0000 7f65cb000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c407ae10 con 0x7f65c4107e60 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.053+0000 7f65c9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.053+0000 7f65c9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33106/0 (socket says 172.21.15.2:33106) 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.053+0000 7f65c9c00700 1 -- 172.21.15.2:0/986424704 learned_addr learned my addr 172.21.15.2:0/986424704 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.054+0000 7f65c9c00700 1 -- 172.21.15.2:0/986424704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c410a790 con 0x7f65c4107e60 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.054+0000 7f65c9c00700 1 --2- 172.21.15.2:0/986424704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098a0 tx=0x7f65b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa602ec777c0581f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.054+0000 7f65c3e00700 1 -- 172.21.15.2:0/986424704 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b8004160 con 0x7f65c4107e60 2024-03-11T18:22:39.109 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.054+0000 7f65c3e00700 1 -- 172.21.15.2:0/986424704 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65b80042c0 con 0x7f65c4107e60 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.054+0000 7f65c3e00700 1 -- 172.21.15.2:0/986424704 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b8011a80 con 0x7f65c4107e60 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 -- 172.21.15.2:0/986424704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 msgr2=0x7f65c410a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 --2- 172.21.15.2:0/986424704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65b80098a0 tx=0x7f65b8009bb0 comp rx=0 tx=0).stop 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 -- 172.21.15.2:0/986424704 shutdown_connections 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 --2- 172.21.15.2:0/986424704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c410a250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 -- 172.21.15.2:0/986424704 >> 172.21.15.2:0/986424704 conn(0x7f65c41018b0 msgr2=0x7f65c4103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 -- 172.21.15.2:0/986424704 shutdown_connections 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.055+0000 7f65cb000700 1 -- 172.21.15.2:0/986424704 wait complete. 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.056+0000 7f65cb000700 1 Processor -- start 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65cb000700 1 -- start start 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65cb000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:39.110 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65cb000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c4107c20 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65c9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65c9c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33110/0 (socket says 172.21.15.2:33110) 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.057+0000 7f65c9c00700 1 -- 172.21.15.2:0/3669160014 learned_addr learned my addr 172.21.15.2:0/3669160014 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65c9c00700 1 -- 172.21.15.2:0/3669160014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b8009550 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65c9c00700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65b8011740 tx=0x7f65b8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b8003b20 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65b801a910 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f65b80248c0 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c4105e10 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.058+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c41062b0 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.060+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f65b801aa80 con 0x7f65c4107e60 2024-03-11T18:22:39.111 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.060+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c410b580 con 0x7f65c4107e60 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.060+0000 7f65c2000700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 0x7f65ac03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.060+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f65b804d2d0 con 0x7f65c4107e60 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.060+0000 7f65c9200700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 0x7f65ac03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.061+0000 7f65c9200700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 0x7f65ac03b310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65b0006fd0 tx=0x7f65b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.066+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f65b801adf0 con 0x7f65c4107e60 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.407+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f65c410b790 con 0x7f65c4107e60 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.413+0000 7f65c2000700 1 -- 172.21.15.2:0/3669160014 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f65b8018be0 con 0x7f65c4107e60 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.415+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 msgr2=0x7f65ac03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.415+0000 7f65cb000700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 0x7f65ac03b310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f65b0006fd0 tx=0x7f65b0006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.415+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 msgr2=0x7f65c41076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.415+0000 7f65cb000700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65b8011740 tx=0x7f65b8011b00 comp rx=0 tx=0).stop 2024-03-11T18:22:39.112 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 shutdown_connections 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f65ac038e50 0x7f65ac03b310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 --2- 172.21.15.2:0/3669160014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65c4107e60 0x7f65c41076e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 >> 172.21.15.2:0/3669160014 conn(0x7f65c41018b0 msgr2=0x7f65c4102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 shutdown_connections 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr 2024-03-11T18:22:38.416+0000 7f65cb000700 1 -- 172.21.15.2:0/3669160014 wait complete. 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T18:22:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: ceph telemetry on 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:For more information see: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:39.114 INFO:teuthology.orchestra.run.smithi002.stdout:Bootstrap complete. 2024-03-11T18:22:39.211 INFO:tasks.cephadm:Fetching config... 2024-03-11T18:22:39.212 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:22:39.212 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T18:22:39.305 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T18:22:39.305 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:22:39.305 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T18:22:39.346 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T18:22:39.346 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:22:39.346 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T18:22:39.475 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T18:22:39.475 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:22:39.475 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T18:22:39.528 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T18:22:39.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T18:22:39.664 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:22:39.664 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:39 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3669160014' entity='client.admin' 2024-03-11T18:22:39.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T18:22:39.769 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:22:39.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T18:22:39.835 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBBsqXMgL1PF6q8gLCuMTw5HLg9opT3xcsuUOEILv5aFUspL+22LME/t7GyCGufbLW31c5fYNXKMsJWZPbti1nAZJwKjLyq7Q3xyK/qNEkp/AxsfinsXVXrv+FMr98iQ+FVcRRUJuCbfstglbtYq8SjQds9KhSx27Xsg/vh1L3py7D1Rd3e8dF5IF2gSkWyMvNZBNcKyNEnvyVsneLwZ52jH79IvO3JGsGefBQJW8LkANFo1CRgyKppqKi16tU9DFpaTWwoOi5GJ0gP2mm7jxQPlaljTHu/UQVy2JoBwTlKphUjrh7qyo6ZCUF0zkERPeP7trHOoLkxOskWMy9IRSLlCNv/1GNnHbUACcc3NPYPiLDPRwKYOl6rJTquzPv6h+KAP8epEHh8wIKyoItlYXJ/a++lKCKYaacNaQ2ot1sHiqrGpb1qd1e17cDy4yaFBRtBHdNciPn0qeepRvixWTH+d+JJB6t6WGk1qGFTs1lsq016e2yxkS5OEeiru5RmfM= ceph-02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:22:39.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T18:22:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 -- 172.21.15.2:0/3629739063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 msgr2=0x7fdc2c101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 --2- 172.21.15.2:0/3629739063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c101ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009910 tx=0x7fdc20009c20 comp rx=0 tx=0).stop 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 -- 172.21.15.2:0/3629739063 shutdown_connections 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 --2- 172.21.15.2:0/3629739063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c101ac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 -- 172.21.15.2:0/3629739063 >> 172.21.15.2:0/3629739063 conn(0x7fdc2c0fcc40 msgr2=0x7fdc2c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.123+0000 7fdc34600700 1 -- 172.21.15.2:0/3629739063 shutdown_connections 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.124+0000 7fdc34600700 1 -- 172.21.15.2:0/3629739063 wait complete. 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc34600700 1 Processor -- start 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc34600700 1 -- start start 2024-03-11T18:22:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc34600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc34600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c075e70 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc31c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc31c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33128/0 (socket says 172.21.15.2:33128) 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.125+0000 7fdc31c00700 1 -- 172.21.15.2:0/1392387764 learned_addr learned my addr 172.21.15.2:0/1392387764 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc31c00700 1 -- 172.21.15.2:0/1392387764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc200095f0 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc31c00700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005850 tx=0x7fdc20004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc2001d070 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc2000be40 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc20022ea0 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.126+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c074060 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.127+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c074500 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.128+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7fdc20022670 con 0x7fdc2c1016a0 2024-03-11T18:22:43.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.128+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c107e30 con 0x7fdc2c1016a0 2024-03-11T18:22:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.128+0000 7fdc2a000700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 0x7fdc1403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.128+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdc2004cee0 con 0x7fdc2c1016a0 2024-03-11T18:22:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.128+0000 7fdc31200700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 0x7fdc1403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.129+0000 7fdc31200700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 0x7fdc1403b210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006fd0 tx=0x7fdc18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:43.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.134+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc2000b960 con 0x7fdc2c1016a0 2024-03-11T18:22:43.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.358+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fdc2c0632a0 con 0x7fdc2c1016a0 2024-03-11T18:22:43.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.363+0000 7fdc2a000700 1 -- 172.21.15.2:0/1392387764 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fdc20027070 con 0x7fdc2c1016a0 2024-03-11T18:22:43.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 msgr2=0x7fdc1403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:43.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 0x7fdc1403b210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdc18006fd0 tx=0x7fdc18006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:43.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 msgr2=0x7fdc2c075930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc20005850 tx=0x7fdc20004ff0 comp rx=0 tx=0).stop 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 shutdown_connections 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fdc14038d50 0x7fdc1403b210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 --2- 172.21.15.2:0/1392387764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc2c1016a0 0x7fdc2c075930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.366+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 >> 172.21.15.2:0/1392387764 conn(0x7fdc2c0fcc40 msgr2=0x7fdc2c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.367+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 shutdown_connections 2024-03-11T18:22:43.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:43.367+0000 7fdc34600700 1 -- 172.21.15.2:0/1392387764 wait complete. 2024-03-11T18:22:44.152 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T18:22:44.152 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T18:22:44.567 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:22:44.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1392387764' entity='client.admin' 2024-03-11T18:22:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:22:46.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:46.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:46.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 -- 172.21.15.2:0/3457331625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 msgr2=0x7fa448101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 --2- 172.21.15.2:0/3457331625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa448101ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa438009910 tx=0x7fa438009c20 comp rx=0 tx=0).stop 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 -- 172.21.15.2:0/3457331625 shutdown_connections 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 --2- 172.21.15.2:0/3457331625 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa448101ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 -- 172.21.15.2:0/3457331625 >> 172.21.15.2:0/3457331625 conn(0x7fa4480fcc60 msgr2=0x7fa4480ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 -- 172.21.15.2:0/3457331625 shutdown_connections 2024-03-11T18:22:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.003+0000 7fa44dc00700 1 -- 172.21.15.2:0/3457331625 wait complete. 2024-03-11T18:22:47.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.005+0000 7fa44dc00700 1 Processor -- start 2024-03-11T18:22:47.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.005+0000 7fa44dc00700 1 -- start start 2024-03-11T18:22:47.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.005+0000 7fa44dc00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:47.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.005+0000 7fa44dc00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa448075e20 con 0x7fa4481016c0 2024-03-11T18:22:47.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.006+0000 7fa446a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:47.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.006+0000 7fa446a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42078/0 (socket says 172.21.15.2:42078) 2024-03-11T18:22:47.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.006+0000 7fa446a00700 1 -- 172.21.15.2:0/2478122126 learned_addr learned my addr 172.21.15.2:0/2478122126 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:47.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.006+0000 7fa446a00700 1 -- 172.21.15.2:0/2478122126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4380095f0 con 0x7fa4481016c0 2024-03-11T18:22:47.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.006+0000 7fa446a00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa438004f40 tx=0x7fa438005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:47.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.007+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa43801d070 con 0x7fa4481016c0 2024-03-11T18:22:47.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.007+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4380044a0 con 0x7fa4481016c0 2024-03-11T18:22:47.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.007+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa438003a40 con 0x7fa4481016c0 2024-03-11T18:22:47.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.007+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa448074060 con 0x7fa4481016c0 2024-03-11T18:22:47.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.007+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa448074500 con 0x7fa4481016c0 2024-03-11T18:22:47.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.008+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7fa438003cb0 con 0x7fa4481016c0 2024-03-11T18:22:47.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.008+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa448107e50 con 0x7fa4481016c0 2024-03-11T18:22:47.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.009+0000 7fa43f400700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 0x7fa42c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:47.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.009+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa43804cbe0 con 0x7fa4481016c0 2024-03-11T18:22:47.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.009+0000 7fa43fe00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 0x7fa42c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:47.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.010+0000 7fa43fe00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 0x7fa42c03b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa430006fd0 tx=0x7fa430006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:47.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.014+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa438029360 con 0x7fa4481016c0 2024-03-11T18:22:47.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.245+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fa448074740 con 0x7fa42c038db0 2024-03-11T18:22:47.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.252+0000 7fa43f400700 1 -- 172.21.15.2:0/2478122126 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa448074740 con 0x7fa42c038db0 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 msgr2=0x7fa42c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 0x7fa42c03b270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa430006fd0 tx=0x7fa430006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 msgr2=0x7fa4480758e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa438004f40 tx=0x7fa438005020 comp rx=0 tx=0).stop 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 shutdown_connections 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa42c038db0 0x7fa42c03b270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:47.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 --2- 172.21.15.2:0/2478122126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa4481016c0 0x7fa4480758e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:47.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 >> 172.21.15.2:0/2478122126 conn(0x7fa4480fcc60 msgr2=0x7fa4480fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:47.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 shutdown_connections 2024-03-11T18:22:47.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:47.255+0000 7fa44dc00700 1 -- 172.21.15.2:0/2478122126 wait complete. 2024-03-11T18:22:48.057 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi023 2024-03-11T18:22:48.057 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:22:48.058 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/etc/ceph/ceph.conf 2024-03-11T18:22:48.076 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:22:48.076 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:22:48.134 INFO:tasks.cephadm:Adding host smithi023 to orchestrator... 2024-03-11T18:22:48.135 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch host add smithi023 2024-03-11T18:22:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:22:48.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:48 smithi002 ceph-mon[31328]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T18:22:49.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:49 smithi002 ceph-mon[31328]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:22:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:49 smithi002 ceph-mon[31328]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:49 smithi002 ceph-mon[31328]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:22:50.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:50 smithi002 ceph-mon[31328]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:22:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:50.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 -- 172.21.15.2:0/1838429379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 msgr2=0x7f72381079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 --2- 172.21.15.2:0/1838429379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f72381079d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7228009910 tx=0x7f7228009c20 comp rx=0 tx=0).stop 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 -- 172.21.15.2:0/1838429379 shutdown_connections 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 --2- 172.21.15.2:0/1838429379 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f72381079d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 -- 172.21.15.2:0/1838429379 >> 172.21.15.2:0/1838429379 conn(0x7f72380fb600 msgr2=0x7f72380fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 -- 172.21.15.2:0/1838429379 shutdown_connections 2024-03-11T18:22:50.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.620+0000 7f723e200700 1 -- 172.21.15.2:0/1838429379 wait complete. 2024-03-11T18:22:50.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.622+0000 7f723e200700 1 Processor -- start 2024-03-11T18:22:50.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.622+0000 7f723e200700 1 -- start start 2024-03-11T18:22:50.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.623+0000 7f723e200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:50.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.623+0000 7f723e200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72381994d0 con 0x7f72381075f0 2024-03-11T18:22:50.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.623+0000 7f723ce00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:50.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.623+0000 7f723ce00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42102/0 (socket says 172.21.15.2:42102) 2024-03-11T18:22:50.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.623+0000 7f723ce00700 1 -- 172.21.15.2:0/2019603807 learned_addr learned my addr 172.21.15.2:0/2019603807 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:50.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f723ce00700 1 -- 172.21.15.2:0/2019603807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72280095f0 con 0x7f72381075f0 2024-03-11T18:22:50.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f723ce00700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72280092e0 tx=0x7f7228004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:50.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f722801d070 con 0x7f72381075f0 2024-03-11T18:22:50.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f722800b6f0 con 0x7f72381075f0 2024-03-11T18:22:50.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72280037f0 con 0x7f72381075f0 2024-03-11T18:22:50.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.624+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7238199730 con 0x7f72381075f0 2024-03-11T18:22:50.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.625+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7238199c30 con 0x7f72381075f0 2024-03-11T18:22:50.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.626+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f7228004260 con 0x7f72381075f0 2024-03-11T18:22:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.626+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72380632a0 con 0x7f72381075f0 2024-03-11T18:22:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.626+0000 7f7236000700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 0x7f721c03b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.626+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f722804c960 con 0x7f72381075f0 2024-03-11T18:22:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.627+0000 7f722fe00700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 0x7f721c03b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:50.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.627+0000 7f722fe00700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 0x7f721c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7220006fd0 tx=0x7f7220006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:50.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.633+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f722800b860 con 0x7f72381075f0 2024-03-11T18:22:50.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:50.861+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "target": ["mon-mgr", ""]}) v1 -- 0x7f7238199c30 con 0x7f721c038db0 2024-03-11T18:22:51.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:51 smithi002 ceph-mon[31328]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:51.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:51 smithi002 ceph-mon[31328]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:22:52.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:52 smithi002 ceph-mon[31328]: Deploying cephadm binary to smithi023 2024-03-11T18:22:53.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:53 smithi002 ceph-mon[31328]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:55.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:55 smithi002 ceph-mon[31328]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:55.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.912+0000 7f7236000700 1 -- 172.21.15.2:0/2019603807 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f7238199c30 con 0x7f721c038db0 2024-03-11T18:22:55.914 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi023' with addr '172.21.15.23' 2024-03-11T18:22:55.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.915+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 msgr2=0x7f721c03b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:55.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.915+0000 7f723e200700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 0x7f721c03b270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7220006fd0 tx=0x7f7220006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:55.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 msgr2=0x7f7238198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:55.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72280092e0 tx=0x7f7228004ff0 comp rx=0 tx=0).stop 2024-03-11T18:22:55.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 shutdown_connections 2024-03-11T18:22:55.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f721c038db0 0x7f721c03b270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:55.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 --2- 172.21.15.2:0/2019603807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72381075f0 0x7f7238198f90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:55.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 >> 172.21.15.2:0/2019603807 conn(0x7f72380fb600 msgr2=0x7f7238102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:55.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 shutdown_connections 2024-03-11T18:22:55.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:55.916+0000 7f723e200700 1 -- 172.21.15.2:0/2019603807 wait complete. 2024-03-11T18:22:56.750 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T18:22:57.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:57.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:56 smithi002 ceph-mon[31328]: Added host smithi023 2024-03-11T18:22:57.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:22:57.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:57.182 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:22:58.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:57 smithi002 ceph-mon[31328]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:22:58.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 -- 172.21.15.2:0/3360832808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 msgr2=0x7f706c0ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 --2- 172.21.15.2:0/3360832808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c0ffd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f705c009910 tx=0x7f705c009c20 comp rx=0 tx=0).stop 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 -- 172.21.15.2:0/3360832808 shutdown_connections 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 --2- 172.21.15.2:0/3360832808 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c0ffd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 -- 172.21.15.2:0/3360832808 >> 172.21.15.2:0/3360832808 conn(0x7f706c0fb3b0 msgr2=0x7f706c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 -- 172.21.15.2:0/3360832808 shutdown_connections 2024-03-11T18:22:59.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.163+0000 7f7071400700 1 -- 172.21.15.2:0/3360832808 wait complete. 2024-03-11T18:22:59.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.164+0000 7f7071400700 1 Processor -- start 2024-03-11T18:22:59.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.164+0000 7f7071400700 1 -- start start 2024-03-11T18:22:59.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.165+0000 7f7071400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:59.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.165+0000 7f7071400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c1996b0 con 0x7f706c0ff970 2024-03-11T18:22:59.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.165+0000 7f706a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:59.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.165+0000 7f706a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40170/0 (socket says 172.21.15.2:40170) 2024-03-11T18:22:59.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.165+0000 7f706a800700 1 -- 172.21.15.2:0/1000002142 learned_addr learned my addr 172.21.15.2:0/1000002142 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:22:59.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f706a800700 1 -- 172.21.15.2:0/1000002142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c0095f0 con 0x7f706c0ff970 2024-03-11T18:22:59.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f706a800700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f705c0092e0 tx=0x7f705c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f705c01d070 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f705c00b750 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f705c0039d0 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c1998b0 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.166+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c199d50 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.167+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c0632a0 con 0x7f706c0ff970 2024-03-11T18:22:59.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.168+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f705c00b8c0 con 0x7f706c0ff970 2024-03-11T18:22:59.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.168+0000 7f7063400700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 0x7f705003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:22:59.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.168+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f705c04caf0 con 0x7f706c0ff970 2024-03-11T18:22:59.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.168+0000 7f7069e00700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 0x7f705003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:22:59.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.169+0000 7f7069e00700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 0x7f705003b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7058006fd0 tx=0x7f7058006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:22:59.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.174+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f705c02ada0 con 0x7f706c0ff970 2024-03-11T18:22:59.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.402+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f706c10dd90 con 0x7f7050038db0 2024-03-11T18:22:59.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.405+0000 7f7063400700 1 -- 172.21.15.2:0/1000002142 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7f706c10dd90 con 0x7f7050038db0 2024-03-11T18:22:59.406 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:22:59.406 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}] 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 msgr2=0x7f705003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 0x7f705003b270 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7058006fd0 tx=0x7f7058006e40 comp rx=0 tx=0).stop 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 msgr2=0x7f706c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f705c0092e0 tx=0x7f705c004ff0 comp rx=0 tx=0).stop 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 shutdown_connections 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7050038db0 0x7f705003b270 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 --2- 172.21.15.2:0/1000002142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f706c0ff970 0x7f706c199170 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.407+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 >> 172.21.15.2:0/1000002142 conn(0x7f706c0fb3b0 msgr2=0x7f706c10a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.408+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 shutdown_connections 2024-03-11T18:22:59.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:22:59.408+0000 7f7071400700 1 -- 172.21.15.2:0/1000002142 wait complete. 2024-03-11T18:23:00.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:59 smithi002 ceph-mon[31328]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:00.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:22:59 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:00.137 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-03-11T18:23:00.137 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:23:00.137 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-03-11T18:23:00.156 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:23:00.156 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:00.214 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-03-11T18:23:00.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch host add smithi165 2024-03-11T18:23:00.676 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:00.984 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:00 smithi002 ceph-mon[31328]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:23:01.849 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:01 smithi002 ceph-mon[31328]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 -- 172.21.15.2:0/2803143933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 msgr2=0x7f7d4c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 --2- 172.21.15.2:0/2803143933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c1082b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009910 tx=0x7f7d3c009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 -- 172.21.15.2:0/2803143933 shutdown_connections 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 --2- 172.21.15.2:0/2803143933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c1082b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 -- 172.21.15.2:0/2803143933 >> 172.21.15.2:0/2803143933 conn(0x7f7d4c077410 msgr2=0x7f7d4c079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:02.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 -- 172.21.15.2:0/2803143933 shutdown_connections 2024-03-11T18:23:02.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.702+0000 7f7d53200700 1 -- 172.21.15.2:0/2803143933 wait complete. 2024-03-11T18:23:02.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.703+0000 7f7d53200700 1 Processor -- start 2024-03-11T18:23:02.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.703+0000 7f7d53200700 1 -- start start 2024-03-11T18:23:02.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.704+0000 7f7d53200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:02.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.704+0000 7f7d53200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c19db20 con 0x7f7d4c107ed0 2024-03-11T18:23:02.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.704+0000 7f7d4be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:02.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.704+0000 7f7d4be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40178/0 (socket says 172.21.15.2:40178) 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.704+0000 7f7d4be00700 1 -- 172.21.15.2:0/2700127286 learned_addr learned my addr 172.21.15.2:0/2700127286 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d4be00700 1 -- 172.21.15.2:0/2700127286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c0095f0 con 0x7f7d4c107ed0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d4be00700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006010 tx=0x7f7d3c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d3c01d070 con 0x7f7d4c107ed0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d3c00bc30 con 0x7f7d4c107ed0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d3c003930 con 0x7f7d4c107ed0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c19dd20 con 0x7f7d4c107ed0 2024-03-11T18:23:02.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.705+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c19e140 con 0x7f7d4c107ed0 2024-03-11T18:23:02.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.707+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f7d3c003ae0 con 0x7f7d4c107ed0 2024-03-11T18:23:02.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.707+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c0632a0 con 0x7f7d4c107ed0 2024-03-11T18:23:02.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.707+0000 7f7d48c00700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 0x7f7d3003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:02.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.707+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d3c04cad0 con 0x7f7d4c107ed0 2024-03-11T18:23:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.708+0000 7f7d4b400700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 0x7f7d3003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:02.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.708+0000 7f7d4b400700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 0x7f7d3003b270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d38006fd0 tx=0x7f7d38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.713+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d3c00b6f0 con 0x7f7d4c107ed0 2024-03-11T18:23:02.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:02.947+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d4c1121e0 con 0x7f7d30038db0 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:23:03.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:03.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:03.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:02 smithi002 ceph-mon[31328]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:05.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:04 smithi002 ceph-mon[31328]: Deploying cephadm binary to smithi165 2024-03-11T18:23:06.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:05 smithi002 ceph-mon[31328]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:07.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.845+0000 7f7d48c00700 1 -- 172.21.15.2:0/2700127286 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7d4c1121e0 con 0x7f7d30038db0 2024-03-11T18:23:07.846 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-03-11T18:23:07.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.848+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 msgr2=0x7f7d3003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:07.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.848+0000 7f7d53200700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 0x7f7d3003b270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7d38006fd0 tx=0x7f7d38006e40 comp rx=0 tx=0).stop 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.848+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 msgr2=0x7f7d4c19d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.848+0000 7f7d53200700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006010 tx=0x7f7d3c004ff0 comp rx=0 tx=0).stop 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 shutdown_connections 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7d30038db0 0x7f7d3003b270 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 --2- 172.21.15.2:0/2700127286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d4c107ed0 0x7f7d4c19d5e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 >> 172.21.15.2:0/2700127286 conn(0x7f7d4c077410 msgr2=0x7f7d4c10ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:07.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 shutdown_connections 2024-03-11T18:23:07.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:07.849+0000 7f7d53200700 1 -- 172.21.15.2:0/2700127286 wait complete. 2024-03-11T18:23:08.192 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:07 smithi002 ceph-mon[31328]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:08.192 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:08.192 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:08.566 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T18:23:09.038 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:08 smithi002 ceph-mon[31328]: Added host smithi165 2024-03-11T18:23:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:08 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:08 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:10.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:09 smithi002 ceph-mon[31328]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 -- 172.21.15.2:0/3850626754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 msgr2=0x7fca70109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 --2- 172.21.15.2:0/3850626754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70109c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca60009910 tx=0x7fca60009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 -- 172.21.15.2:0/3850626754 shutdown_connections 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 --2- 172.21.15.2:0/3850626754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70109c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 -- 172.21.15.2:0/3850626754 >> 172.21.15.2:0/3850626754 conn(0x7fca700ff320 msgr2=0x7fca70101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:11.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 -- 172.21.15.2:0/3850626754 shutdown_connections 2024-03-11T18:23:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.071+0000 7fca74c00700 1 -- 172.21.15.2:0/3850626754 wait complete. 2024-03-11T18:23:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.073+0000 7fca74c00700 1 Processor -- start 2024-03-11T18:23:11.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.073+0000 7fca74c00700 1 -- start start 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.073+0000 7fca74c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.073+0000 7fca74c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7007a0e0 con 0x7fca70109850 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.074+0000 7fca6de00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.074+0000 7fca6de00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50808/0 (socket says 172.21.15.2:50808) 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.074+0000 7fca6de00700 1 -- 172.21.15.2:0/3811579731 learned_addr learned my addr 172.21.15.2:0/3811579731 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.074+0000 7fca6de00700 1 -- 172.21.15.2:0/3811579731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600095f0 con 0x7fca70109850 2024-03-11T18:23:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca6de00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fca60006010 tx=0x7fca60004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca6001d070 con 0x7fca70109850 2024-03-11T18:23:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca74c00700 1 -- 172.21.15.2:0/3811579731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca700766c0 con 0x7fca70109850 2024-03-11T18:23:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca6000bbd0 con 0x7fca70109850 2024-03-11T18:23:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca60003880 con 0x7fca70109850 2024-03-11T18:23:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.075+0000 7fca74c00700 1 -- 172.21.15.2:0/3811579731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca70076b40 con 0x7fca70109850 2024-03-11T18:23:11.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.077+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48005330 con 0x7fca70109850 2024-03-11T18:23:11.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.078+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7fca6000bd40 con 0x7fca70109850 2024-03-11T18:23:11.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.078+0000 7fca66a00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 0x7fca5403b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:11.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.078+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca6004cad0 con 0x7fca70109850 2024-03-11T18:23:11.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.078+0000 7fca6d400700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 0x7fca5403b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:11.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.079+0000 7fca6d400700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 0x7fca5403b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fca5c006fd0 tx=0x7fca5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:11.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.087+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fca6002ac80 con 0x7fca70109850 2024-03-11T18:23:11.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.322+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca48000bc0 con 0x7fca54038d50 2024-03-11T18:23:11.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.325+0000 7fca66a00700 1 -- 172.21.15.2:0/3811579731 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7fca48000bc0 con 0x7fca54038d50 2024-03-11T18:23:11.327 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:23:11.327 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-03-11T18:23:11.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.327+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 msgr2=0x7fca5403b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:11.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.327+0000 7fca5be00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 0x7fca5403b210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fca5c006fd0 tx=0x7fca5c006e40 comp rx=0 tx=0).stop 2024-03-11T18:23:11.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.327+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 msgr2=0x7fca70076180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:11.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.327+0000 7fca5be00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fca60006010 tx=0x7fca60004ff0 comp rx=0 tx=0).stop 2024-03-11T18:23:11.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 shutdown_connections 2024-03-11T18:23:11.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca54038d50 0x7fca5403b210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:11.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 --2- 172.21.15.2:0/3811579731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca70109850 0x7fca70076180 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:11.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 >> 172.21.15.2:0/3811579731 conn(0x7fca700ff320 msgr2=0x7fca7010b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:11.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 shutdown_connections 2024-03-11T18:23:11.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:11.328+0000 7fca5be00700 1 -- 172.21.15.2:0/3811579731 wait complete. 2024-03-11T18:23:11.917 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:11 smithi002 ceph-mon[31328]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:11.917 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:11.999 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T18:23:11.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T18:23:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:12.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:12 smithi002 ceph-mon[31328]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:23:13.923 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:13 smithi002 ceph-mon[31328]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.461+0000 7fe469600700 1 -- 172.21.15.2:0/4143704007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe45c0054c0 con 0x7fe46c107650 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 -- 172.21.15.2:0/4143704007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 msgr2=0x7fe46c107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 --2- 172.21.15.2:0/4143704007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c107a30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009910 tx=0x7fe45c009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 -- 172.21.15.2:0/4143704007 shutdown_connections 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 --2- 172.21.15.2:0/4143704007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c107a30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 -- 172.21.15.2:0/4143704007 >> 172.21.15.2:0/4143704007 conn(0x7fe46c0fd180 msgr2=0x7fe46c0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:14.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 -- 172.21.15.2:0/4143704007 shutdown_connections 2024-03-11T18:23:14.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.462+0000 7fe471a00700 1 -- 172.21.15.2:0/4143704007 wait complete. 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.463+0000 7fe471a00700 1 Processor -- start 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe471a00700 1 -- start start 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe471a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe471a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c1996c0 con 0x7fe46c107650 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe46aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe46aa00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50826/0 (socket says 172.21.15.2:50826) 2024-03-11T18:23:14.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.464+0000 7fe46aa00700 1 -- 172.21.15.2:0/2577839521 learned_addr learned my addr 172.21.15.2:0/2577839521 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.465+0000 7fe46aa00700 1 -- 172.21.15.2:0/2577839521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c0095f0 con 0x7fe46c107650 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.465+0000 7fe46aa00700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe45c004f40 tx=0x7fe45c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.465+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe45c01d070 con 0x7fe46c107650 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.465+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c199920 con 0x7fe46c107650 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.466+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46c199da0 con 0x7fe46c107650 2024-03-11T18:23:14.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.466+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe45c00bbd0 con 0x7fe46c107650 2024-03-11T18:23:14.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.466+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe45c0038a0 con 0x7fe46c107650 2024-03-11T18:23:14.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.467+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7fe45c003a00 con 0x7fe46c107650 2024-03-11T18:23:14.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.467+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44c005360 con 0x7fe46c107650 2024-03-11T18:23:14.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.467+0000 7fe463400700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 0x7fe45003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:14.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.467+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe45c04c950 con 0x7fe46c107650 2024-03-11T18:23:14.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.468+0000 7fe46a000700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 0x7fe45003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:14.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.468+0000 7fe46a000700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 0x7fe45003b270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe458006fd0 tx=0x7fe458006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:14.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.474+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe45c02a590 con 0x7fe46c107650 2024-03-11T18:23:14.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.706+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe44c005610 con 0x7fe46c107650 2024-03-11T18:23:14.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.786+0000 7fe463400700 1 -- 172.21.15.2:0/2577839521 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe45c026070 con 0x7fe46c107650 2024-03-11T18:23:14.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 msgr2=0x7fe45003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:14.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 0x7fe45003b270 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe458006fd0 tx=0x7fe458006e40 comp rx=0 tx=0).stop 2024-03-11T18:23:14.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 msgr2=0x7fe46c199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe45c004f40 tx=0x7fe45c005020 comp rx=0 tx=0).stop 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 shutdown_connections 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe450038db0 0x7fe45003b270 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 --2- 172.21.15.2:0/2577839521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe46c107650 0x7fe46c199180 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 >> 172.21.15.2:0/2577839521 conn(0x7fe46c0fd180 msgr2=0x7fe46c1089b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.790+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 shutdown_connections 2024-03-11T18:23:14.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:14.791+0000 7fe471a00700 1 -- 172.21.15.2:0/2577839521 wait complete. 2024-03-11T18:23:14.794 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T18:23:15.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:14 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2577839521' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T18:23:15.545 INFO:tasks.cephadm:Adding mon.a on smithi002 2024-03-11T18:23:15.545 INFO:tasks.cephadm:Adding mon.b on smithi023 2024-03-11T18:23:15.545 INFO:tasks.cephadm:Adding mon.c on smithi165 2024-03-11T18:23:15.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c' 2024-03-11T18:23:15.770 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2577839521' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:16.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/1718384745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 msgr2=0x7f9d88107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/1718384745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88107a40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009910 tx=0x7f9d7c009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/1718384745 shutdown_connections 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/1718384745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88107a40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/1718384745 >> 172.21.15.165:0/1718384745 conn(0x7f9d88076c20 msgr2=0x7f9d88077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/1718384745 shutdown_connections 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.035+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/1718384745 wait complete. 2024-03-11T18:23:17.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8eaa0700 1 Processor -- start 2024-03-11T18:23:17.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8eaa0700 1 -- start start 2024-03-11T18:23:17.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8eaa0700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:17.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8eaa0700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d88102cb0 con 0x7f9d88107660 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8c83c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8c83c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.165:44176/0 (socket says 172.21.15.165:44176) 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.036+0000 7f9d8c83c700 1 -- 172.21.15.165:0/3035159555 learned_addr learned my addr 172.21.15.165:0/3035159555 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.037+0000 7f9d8c83c700 1 -- 172.21.15.165:0/3035159555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0095f0 con 0x7f9d88107660 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.037+0000 7f9d8c83c700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004f40 tx=0x7f9d7c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.038+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d7c01d070 con 0x7f9d88107660 2024-03-11T18:23:17.039 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.038+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88101350 con 0x7f9d88107660 2024-03-11T18:23:17.040 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.038+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d881017d0 con 0x7f9d88107660 2024-03-11T18:23:17.040 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.038+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d7c00bbd0 con 0x7f9d88107660 2024-03-11T18:23:17.040 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.038+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d7c0038a0 con 0x7f9d88107660 2024-03-11T18:23:17.040 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.039+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c005330 con 0x7f9d88107660 2024-03-11T18:23:17.040 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.039+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f9d7c00bd40 con 0x7f9d88107660 2024-03-11T18:23:17.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.040+0000 7f9d85ffb700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 0x7f9d7803b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:17.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.040+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d7c04c970 con 0x7f9d88107660 2024-03-11T18:23:17.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.040+0000 7f9d87fff700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 0x7f9d7803b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:17.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.041+0000 7f9d87fff700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 0x7f9d7803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006fd0 tx=0x7f9d74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:17.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.055+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d7c00b6f0 con 0x7f9d88107660 2024-03-11T18:23:17.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.261+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d6c000c70 con 0x7f9d78038d50 2024-03-11T18:23:17.277 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.277+0000 7f9d85ffb700 1 -- 172.21.15.165:0/3035159555 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9d6c000c70 con 0x7f9d78038d50 2024-03-11T18:23:17.278 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-03-11T18:23:17.280 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 msgr2=0x7f9d7803b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:17.280 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 0x7f9d7803b210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006fd0 tx=0x7f9d74006e40 comp rx=0 tx=0).stop 2024-03-11T18:23:17.280 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 msgr2=0x7f9d88100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004f40 tx=0x7f9d7c005020 comp rx=0 tx=0).stop 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 shutdown_connections 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9d78038d50 0x7f9d7803b210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 --2- 172.21.15.165:0/3035159555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d88107660 0x7f9d88100e10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 >> 172.21.15.165:0/3035159555 conn(0x7f9d88076c20 msgr2=0x7f9d880fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 shutdown_connections 2024-03-11T18:23:17.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:17.278+0000 7f9d8eaa0700 1 -- 172.21.15.165:0/3035159555 wait complete. 2024-03-11T18:23:17.912 DEBUG:teuthology.orchestra.run.smithi023:mon.b> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T18:23:17.915 DEBUG:teuthology.orchestra.run.smithi165:mon.c> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T18:23:17.917 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T18:23:17.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:18.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:19.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:18 smithi002 ceph-mon[31328]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:19.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:18 smithi002 ceph-mon[31328]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c;count:3 2024-03-11T18:23:19.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:18 smithi002 ceph-mon[31328]: Deploying daemon mon.c on smithi165 2024-03-11T18:23:19.433 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:20.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:19 smithi002 ceph-mon[31328]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:21.080 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:20 smithi165 systemd[1]: Starting Ceph mon.c for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:23:21.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 -- 172.21.15.165:0/1556647702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 msgr2=0x7f6d20106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:21.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1556647702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d20106a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6d10009910 tx=0x7f6d10009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:21.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 -- 172.21.15.165:0/1556647702 shutdown_connections 2024-03-11T18:23:21.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1556647702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d20106a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:21.499 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 -- 172.21.15.165:0/1556647702 >> 172.21.15.165:0/1556647702 conn(0x7f6d200fc040 msgr2=0x7f6d200fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 -- 172.21.15.165:0/1556647702 shutdown_connections 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.492+0000 7f6d24d16700 1 -- 172.21.15.165:0/1556647702 wait complete. 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.493+0000 7f6d24d16700 1 Processor -- start 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.493+0000 7f6d24d16700 1 -- start start 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.493+0000 7f6d24d16700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.493+0000 7f6d24d16700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2019ae30 con 0x7f6d20106620 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.493+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.494+0000 7f6d1e59c700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.165:58200/0 (socket says 172.21.15.165:58200) 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.494+0000 7f6d1e59c700 1 -- 172.21.15.165:0/1890292718 learned_addr learned my addr 172.21.15.165:0/1890292718 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:23:21.500 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.494+0000 7f6d1e59c700 1 -- 172.21.15.165:0/1890292718 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100095f0 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.494+0000 7f6d1e59c700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6d10004f40 tx=0x7f6d10005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.495+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d1001d070 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.495+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2019b090 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.495+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d20194ab0 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.495+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d1000bc30 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.495+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d10003910 con 0x7f6d20106620 2024-03-11T18:23:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.496+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d00005330 con 0x7f6d20106620 2024-03-11T18:23:21.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.496+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f6d1000b6f0 con 0x7f6d20106620 2024-03-11T18:23:21.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.496+0000 7f6d0f7fe700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 0x7f6d0803d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:21.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.497+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d1004ce50 con 0x7f6d20106620 2024-03-11T18:23:21.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.497+0000 7f6d1dd9b700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 0x7f6d0803d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:21.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.498+0000 7f6d1dd9b700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 0x7f6d0803d470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:21.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.512+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d10003a70 con 0x7f6d20106620 2024-03-11T18:23:21.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.794+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d000058b0 con 0x7f6d20106620 2024-03-11T18:23:21.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.795+0000 7f6d0f7fe700 1 -- 172.21.15.165:0/1890292718 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+741 (secure 0 0 0) 0x7f6d10026020 con 0x7f6d20106620 2024-03-11T18:23:21.796 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:23:21.796 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","modified":"2024-03-11T18:21:15.790259Z","created":"2024-03-11T18:21:15.790259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T18:23:21.798 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 msgr2=0x7f6d0803d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 0x7f6d0803d470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006fd0 tx=0x7f6d14006e40 comp rx=0 tx=0).stop 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 msgr2=0x7f6d2019a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6d10004f40 tx=0x7f6d10005020 comp rx=0 tx=0).stop 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 shutdown_connections 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d0803afb0 0x7f6d0803d470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 --2- 172.21.15.165:0/1890292718 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d20106620 0x7f6d2019a8f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 >> 172.21.15.165:0/1890292718 conn(0x7f6d200fc040 msgr2=0x7f6d200fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 shutdown_connections 2024-03-11T18:23:21.799 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:21.797+0000 7f6d24d16700 1 -- 172.21.15.165:0/1890292718 wait complete. 2024-03-11T18:23:21.800 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-03-11T18:23:21.894 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 podman[36599]: 2024-03-11 18:23:21.641375448 +0000 UTC m=+0.112900826 container create 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T18:23:21.894 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 podman[36599]: 2024-03-11 18:23:21.546125393 +0000 UTC m=+0.017650783 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:23:21.895 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 podman[36599]: 2024-03-11 18:23:21.894009931 +0000 UTC m=+0.365535308 container init 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397) 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 podman[36599]: 2024-03-11 18:23:21.898134381 +0000 UTC m=+0.369659749 container start 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 2 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pidfile_write: ignore empty --pid-file 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: load: jerasure load: lrc 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: RocksDB version: 7.9.2 2024-03-11T18:23:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Git sha 0 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: DB SUMMARY 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: DB Session ID: SRF8KW7L0B8YWPGL79MM 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: CURRENT file: CURRENT 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.error_if_exists: 0 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.create_if_missing: 0 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.paranoid_checks: 1 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T18:23:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.env: 0x55c8ed64eae0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.info_log: 0x55c8ef17d8c0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.statistics: (nil) 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.use_fsync: 0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_log_file_size: 0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T18:23:22.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_fallocate: 1 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.use_direct_reads: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.db_log_dir: 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.wal_dir: 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T18:23:22.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.write_buffer_manager: 0x55c8ef151400 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T18:23:22.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.unordered_write: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.row_cache: None 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.wal_filter: None 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.two_write_queues: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.wal_compression: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.atomic_flush: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T18:23:22.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T18:23:22.161 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T18:23:22.162 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.log_readahead_size: 0 2024-03-11T18:23:22.162 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_background_jobs: 2 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_background_compactions: -1 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_subcompactions: 1 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T18:23:22.163 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_open_files: -1 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_background_flushes: -1 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Compression algorithms supported: 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kZSTD supported: 0 2024-03-11T18:23:22.164 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kXpressCompression supported: 0 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kZlibCompression supported: 1 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kSnappyCompression supported: 1 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kLZ4Compression supported: 1 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: kBZip2Compression supported: 0 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.merge_operator: 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_filter: None 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T18:23:22.165 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8ef17d9c0) 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: cache_index_and_filter_blocks: 1 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: pin_top_level_index_and_filter: 1 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_type: 0 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: data_block_index_type: 0 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_shortening: 1 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: checksum: 4 2024-03-11T18:23:22.166 INFO:journalctl@ceph.mon.c.smithi165.stdout: no_block_cache: 0 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache: 0x55c8ef181090 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_name: BinnedLRUCache 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_options: 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: capacity : 536870912 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: num_shard_bits : 4 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: strict_capacity_limit : 0 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: high_pri_pool_ratio: 0.000 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_cache_compressed: (nil) 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: persistent_cache: (nil) 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_size: 4096 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_size_deviation: 10 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_restart_interval: 16 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: index_block_restart_interval: 1 2024-03-11T18:23:22.167 INFO:journalctl@ceph.mon.c.smithi165.stdout: metadata_block_size: 4096 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: partition_filters: 0 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: use_delta_encoding: 1 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: filter_policy: bloomfilter 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: whole_key_filtering: 1 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: verify_compression: 0 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: read_amp_bytes_per_bit: 0 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: format_version: 5 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: enable_index_compression: 1 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: block_align: 0 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: max_auto_readahead_size: 262144 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: prepopulate_block_cache: 0 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: initial_auto_readahead_size: 8192 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T18:23:22.168 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression: NoCompression 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.num_levels: 7 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T18:23:22.169 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T18:23:22.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T18:23:22.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T18:23:22.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T18:23:22.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T18:23:22.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T18:23:22.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T18:23:22.173 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T18:23:22.173 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T18:23:22.174 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T18:23:22.175 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.table_properties_collectors: 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.inplace_update_support: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.bloom_locality: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.max_successive_merges: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.ttl: 2592000 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T18:23:22.176 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enable_blob_files: false 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.min_blob_size: 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T18:23:22.177 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 02bf3599-f5b9-435a-a413-216c8ba0554c 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181401927500, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181401927913, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710181401, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "02bf3599-f5b9-435a-a413-216c8ba0554c", "db_session_id": "SRF8KW7L0B8YWPGL79MM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710181401927993, "job": 1, "event": "recovery_finished"} 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8ef272000 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: DB pointer 0x55c8ef25c000 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** DB Stats ** 2024-03-11T18:23:22.178 INFO:journalctl@ceph.mon.c.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T18:23:22.179 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** Compaction Stats [default] ** 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T18:23:22.180 INFO:journalctl@ceph.mon.c.smithi165.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: Block cache BinnedLRUCache@0x55c8ef181090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: using public_addr v2:172.21.15.165:0/0 -> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-03-11T18:23:22.181 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: starting mon.c rank -1 at public addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] at bind addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(???) e0 preinit fsid 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: e1 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: legacy client fscid: -1 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout: No filesystems configured 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T18:23:22.182 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:22.183 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Deploying cephadm binary to smithi023 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Added host smithi023 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.184 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.185 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Deploying cephadm binary to smithi165 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Added host smithi165 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.186 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2577839521' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:22.187 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2577839521' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:22.188 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c;count:3 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: Deploying daemon mon.c on smithi165 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/1890292718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:22.189 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:21 smithi165 ceph-mon[36634]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T18:23:22.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:21 smithi002 ceph-mon[31328]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:22.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:21 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/1890292718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:22.622 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:22 smithi165 bash[36599]: 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb 2024-03-11T18:23:22.623 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:22 smithi165 systemd[1]: Started Ceph mon.c for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:23:23.650 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T18:23:23.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T18:23:23.896 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:23:27.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:23:27.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: mon.a calling monitor election 2024-03-11T18:23:27.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: mon.c calling monitor election 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: fsmap 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: mgrmap e13: x(active, since 58s) 2024-03-11T18:23:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: overall HEALTH_OK 2024-03-11T18:23:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: mon.a calling monitor election 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: mon.c calling monitor election 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: fsmap 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: mgrmap e13: x(active, since 58s) 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: overall HEALTH_OK 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:28.199 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:28.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:28.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:28.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:28 smithi002 ceph-mon[31328]: Deploying daemon mon.b on smithi023 2024-03-11T18:23:28.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:28.368 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 -- 172.21.15.165:0/4183238012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 msgr2=0x7f3900005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:28.368 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 --2- 172.21.15.165:0/4183238012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3900005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3918100570 tx=0x7f391000feb0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.368 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 -- 172.21.15.165:0/4183238012 shutdown_connections 2024-03-11T18:23:28.368 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 --2- 172.21.15.165:0/4183238012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3900005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.368 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 -- 172.21.15.165:0/4183238012 >> 172.21.15.165:0/4183238012 conn(0x7f39180fc110 msgr2=0x7f39180fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:28.369 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.366+0000 7f391de32700 1 -- 172.21.15.165:0/4183238012 shutdown_connections 2024-03-11T18:23:28.369 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.367+0000 7f391de32700 1 -- 172.21.15.165:0/4183238012 wait complete. 2024-03-11T18:23:28.369 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:28.369 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:28.369 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:28.369 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:28 smithi165 ceph-mon[36634]: Deploying daemon mon.b on smithi023 2024-03-11T18:23:28.369 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:28.371 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.367+0000 7f391de32700 1 Processor -- start 2024-03-11T18:23:28.371 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f391de32700 1 -- start start 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f391de32700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3918197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f391de32700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f391de32700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918197bf0 con 0x7f39000035c0 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f391de32700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3918197d60 con 0x7f3918197660 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56052/0 (socket says 172.21.15.165:56052) 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 -- 172.21.15.165:0/3455203075 learned_addr learned my addr 172.21.15.165:0/3455203075 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 -- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 msgr2=0x7f391819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 -- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 msgr2=0x7f391819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.368+0000 7f3916ffd700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.369+0000 7f39177fe700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3918197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.369+0000 7f39177fe700 1 -- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 msgr2=0x7f391819b9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:23:28.372 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.369+0000 7f39177fe700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.369+0000 7f39177fe700 1 -- 172.21.15.165:0/3455203075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391000f900 con 0x7f39000035c0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f39177fe700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3918197120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3910000c00 tx=0x7f3910005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f3910013960 con 0x7f39000035c0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391819bf90 con 0x7f39000035c0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3910004030 con 0x7f39000035c0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f3910003d00 con 0x7f39000035c0 2024-03-11T18:23:28.373 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.370+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391819c4e0 con 0x7f39000035c0 2024-03-11T18:23:28.374 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.371+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39180632a0 con 0x7f39000035c0 2024-03-11T18:23:28.374 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.372+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f39100041a0 con 0x7f39000035c0 2024-03-11T18:23:28.374 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.372+0000 7f3914ff9700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 0x7f390403b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:28.374 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.372+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f391004e080 con 0x7f39000035c0 2024-03-11T18:23:28.374 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.372+0000 7f3916ffd700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 0x7f390403b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:28.375 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.373+0000 7f3916ffd700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 0x7f390403b020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f390c009180 tx=0x7f390c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:28.389 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.388+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3910034530 con 0x7f39000035c0 2024-03-11T18:23:28.660 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.659+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39181084f0 con 0x7f39000035c0 2024-03-11T18:23:28.661 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.660+0000 7f3914ff9700 1 -- 172.21.15.165:0/3455203075 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1011 (secure 0 0 0) 0x7f3910004550 con 0x7f39000035c0 2024-03-11T18:23:28.662 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:23:28.662 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","modified":"2024-03-11T18:23:21.964622Z","created":"2024-03-11T18:21:15.790259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 msgr2=0x7f390403b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 0x7f390403b020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f390c009180 tx=0x7f390c008040 comp rx=0 tx=0).stop 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 msgr2=0x7f3918197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3918197120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3910000c00 tx=0x7f3910005760 comp rx=0 tx=0).stop 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 shutdown_connections 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39000035c0 0x7f3918197120 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3904038b60 0x7f390403b020 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 --2- 172.21.15.165:0/3455203075 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3918197660 0x7f391819b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 >> 172.21.15.165:0/3455203075 conn(0x7f39180fc110 msgr2=0x7f391806a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 shutdown_connections 2024-03-11T18:23:28.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:28.662+0000 7f391de32700 1 -- 172.21.15.165:0/3455203075 wait complete. 2024-03-11T18:23:28.665 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-03-11T18:23:29.199 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:29 smithi165 ceph-mon[36634]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:29.200 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:29 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/3455203075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:29 smithi002 ceph-mon[31328]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:29 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/3455203075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:30.225 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T18:23:30.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T18:23:30.461 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:23:31.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:31 smithi165 ceph-mon[36634]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:31.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:31 smithi023 systemd[1]: Started Ceph mon.b for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:23:31.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:31 smithi002 ceph-mon[31328]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 -- 172.21.15.165:0/1167689048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60000035c0 msgr2=0x7f6000005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 --2- 172.21.15.165:0/1167689048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60000035c0 0x7f6000005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6014101680 tx=0x7f600c009e80 comp rx=0 tx=0).stop 2024-03-11T18:23:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 -- 172.21.15.165:0/1167689048 shutdown_connections 2024-03-11T18:23:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 --2- 172.21.15.165:0/1167689048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60000035c0 0x7f6000005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 -- 172.21.15.165:0/1167689048 >> 172.21.15.165:0/1167689048 conn(0x7f60140fd260 msgr2=0x7f60140ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:36.348 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 -- 172.21.15.165:0/1167689048 shutdown_connections 2024-03-11T18:23:36.348 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.346+0000 7f601ab98700 1 -- 172.21.15.165:0/1167689048 wait complete. 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 Processor -- start 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 -- start start 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 0x7f601419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6014199f80 con 0x7f6014199890 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60141a5ed0 con 0x7f60000035c0 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f601ab98700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60141a6010 con 0x7f601419e160 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f6019135700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:36.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f6019135700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56054/0 (socket says 172.21.15.165:56054) 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f6019135700 1 -- 172.21.15.165:0/103969057 learned_addr learned my addr 172.21.15.165:0/103969057 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f6013fff700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 0x7f601419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.347+0000 7f6018934700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6019135700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 msgr2=0x7f601419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6019135700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 msgr2=0x7f601419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6019135700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6019135700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6018934700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 msgr2=0x7f6014199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6018934700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 msgr2=0x7f6014199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6018934700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6018934700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:23:36.352 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 msgr2=0x7f601419e610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 msgr2=0x7f6014199350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 -- 172.21.15.165:0/103969057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c0098d0 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.348+0000 7f6013fff700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 0x7f601419dc20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f600400b550 tx=0x7f600400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.349+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004004230 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.349+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60141a62f0 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.349+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6004004390 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.349+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004011b00 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.349+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60141a67f0 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.350+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f6004011c60 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.350+0000 7f6011ffb700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 0x7f5ff403b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.350+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6014106a00 con 0x7f6014199890 2024-03-11T18:23:36.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.350+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f600404b3f0 con 0x7f6014199890 2024-03-11T18:23:36.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.353+0000 7f6018934700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 0x7f5ff403b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:36.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.354+0000 7f6018934700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 0x7f5ff403b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f600c009160 tx=0x7f600c0090b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:36.358 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.358+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6004004500 con 0x7f6014199890 2024-03-11T18:23:36.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: mon.c calling monitor election 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: mon.a calling monitor election 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T18:23:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: fsmap 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: mgrmap e13: x(active, since 68s) 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: mon.b (rank 2) addr [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] is down (out of quorum) 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:36.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:36.642 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.642+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60141095a0 con 0x7f6014199890 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: mon.c calling monitor election 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: mon.a calling monitor election 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:36.643 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: fsmap 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: mgrmap e13: x(active, since 68s) 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: mon.b (rank 2) addr [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] is down (out of quorum) 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:36.644 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:36.645 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.643+0000 7f6011ffb700 1 -- 172.21.15.165:0/103969057 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1275 (secure 0 0 0) 0x7f600403e090 con 0x7f6014199890 2024-03-11T18:23:36.645 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-03-11T18:23:36.645 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","modified":"2024-03-11T18:23:31.148515Z","created":"2024-03-11T18:21:15.790259Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 msgr2=0x7f5ff403b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 0x7f5ff403b6a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f600c009160 tx=0x7f600c0090b0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 msgr2=0x7f601419dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 0x7f601419dc20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f600400b550 tx=0x7f600400b910 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 shutdown_connections 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5ff40391e0 0x7f5ff403b6a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f601419e160 0x7f601419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60000035c0 0x7f6014199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 --2- 172.21.15.165:0/103969057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6014199890 0x7f601419dc20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:36.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 >> 172.21.15.165:0/103969057 conn(0x7f60140fd260 msgr2=0x7f6014074980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:36.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 shutdown_connections 2024-03-11T18:23:36.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:36.646+0000 7f601ab98700 1 -- 172.21.15.165:0/103969057 wait complete. 2024-03-11T18:23:36.648 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-03-11T18:23:37.164 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T18:23:37.165 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T18:23:37.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:37.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:37.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:37.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:37.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/103969057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:37.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:37.587 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:37.588 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:37.588 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:37.588 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:37.588 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/103969057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:37.588 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:37.610 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:38.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: mon.c calling monitor election 2024-03-11T18:23:38.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:23:38.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: mon.a calling monitor election 2024-03-11T18:23:38.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: fsmap 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: mgrmap e13: x(active, since 68s) 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: mon.b (rank 2) addr [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] is down (out of quorum) 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:38.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:38.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:38.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:38.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/103969057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T18:23:38.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:38 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:39.195 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mon.b calling monitor election 2024-03-11T18:23:39.195 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:39.195 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mon.b calling monitor election 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mon.a calling monitor election 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mon.c calling monitor election 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: fsmap 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: mgrmap e13: x(active, since 70s) 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: Cluster is now healthy 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: overall HEALTH_OK 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.196 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.197 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.197 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.197 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.197 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mon.b calling monitor election 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mon.b calling monitor election 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mon.a calling monitor election 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mon.c calling monitor election 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T18:23:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: fsmap 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: mgrmap e13: x(active, since 70s) 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: Cluster is now healthy 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: overall HEALTH_OK 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mon.b calling monitor election 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mon.b calling monitor election 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mon.a calling monitor election 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mon.c calling monitor election 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T18:23:39.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: fsmap 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: mgrmap e13: x(active, since 70s) 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: Cluster is now healthy 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: overall HEALTH_OK 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:23:39.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:23:39.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 -- 172.21.15.2:0/397700891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102800 msgr2=0x7ff4bc102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/397700891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102800 0x7ff4bc102be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009910 tx=0x7ff4ac009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 -- 172.21.15.2:0/397700891 shutdown_connections 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/397700891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102800 0x7ff4bc102be0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 -- 172.21.15.2:0/397700891 >> 172.21.15.2:0/397700891 conn(0x7ff4bc0771b0 msgr2=0x7ff4bc0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 -- 172.21.15.2:0/397700891 shutdown_connections 2024-03-11T18:23:39.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.640+0000 7ff4c1e00700 1 -- 172.21.15.2:0/397700891 wait complete. 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.641+0000 7ff4c1e00700 1 Processor -- start 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.641+0000 7ff4c1e00700 1 -- start start 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 0x7ff4bc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc19c010 0x7ff4bc19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc197e30 con 0x7ff4bc197740 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1a3d80 con 0x7ff4bc102800 2024-03-11T18:23:39.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4c1e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1a3ec0 con 0x7ff4bc19c010 2024-03-11T18:23:39.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4baa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:39.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4ba000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 0x7ff4bc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4baa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47138/0 (socket says 172.21.15.2:47138) 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4baa00700 1 -- 172.21.15.2:0/746626073 learned_addr learned my addr 172.21.15.2:0/746626073 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.642+0000 7ff4bb400700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc19c010 0x7ff4bc19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4ba000700 1 -- 172.21.15.2:0/746626073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc19c010 msgr2=0x7ff4bc19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4ba000700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc19c010 0x7ff4bc19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4ba000700 1 -- 172.21.15.2:0/746626073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 msgr2=0x7ff4bc197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4ba000700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4ba000700 1 -- 172.21.15.2:0/746626073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0095f0 con 0x7ff4bc197740 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.643+0000 7ff4baa00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:23:39.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4ba000700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 0x7ff4bc19bad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b550 tx=0x7ff4a800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8004230 con 0x7ff4bc197740 2024-03-11T18:23:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4a8004390 con 0x7ff4bc197740 2024-03-11T18:23:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8011a90 con 0x7ff4bc197740 2024-03-11T18:23:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc1a41a0 con 0x7ff4bc197740 2024-03-11T18:23:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.644+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc1a4650 con 0x7ff4bc197740 2024-03-11T18:23:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.645+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc069940 con 0x7ff4bc197740 2024-03-11T18:23:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.646+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7ff4a8011c30 con 0x7ff4bc197740 2024-03-11T18:23:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.646+0000 7ff4c0a00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 0x7ff4a003b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.646+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff4a804b230 con 0x7ff4bc197740 2024-03-11T18:23:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.647+0000 7ff4baa00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 0x7ff4a003b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.647+0000 7ff4baa00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 0x7ff4a003b6a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005850 tx=0x7ff4ac00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:39.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.652+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4a8004500 con 0x7ff4bc197740 2024-03-11T18:23:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.877+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff4bc10bae0 con 0x7ff4bc197740 2024-03-11T18:23:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.878+0000 7ff4c0a00700 1 -- 172.21.15.2:0/746626073 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7ff4a800d1c0 con 0x7ff4bc197740 2024-03-11T18:23:39.879 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:23:39.879 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-03-11T18:23:39.880 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:23:39.880 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-03-11T18:23:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 msgr2=0x7ff4a003b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 0x7ff4a003b6a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005850 tx=0x7ff4ac00b540 comp rx=0 tx=0).stop 2024-03-11T18:23:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 msgr2=0x7ff4bc19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 0x7ff4bc19bad0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b550 tx=0x7ff4a800b860 comp rx=0 tx=0).stop 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 shutdown_connections 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00391e0 0x7ff4a003b6a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc19c010 0x7ff4bc19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102800 0x7ff4bc197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 --2- 172.21.15.2:0/746626073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc197740 0x7ff4bc19bad0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 >> 172.21.15.2:0/746626073 conn(0x7ff4bc0771b0 msgr2=0x7ff4bc1036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 shutdown_connections 2024-03-11T18:23:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:39.881+0000 7ff4c1e00700 1 -- 172.21.15.2:0/746626073 wait complete. 2024-03-11T18:23:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: Reconfiguring daemon mon.a on smithi002 2024-03-11T18:23:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:40 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/746626073' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: Reconfiguring daemon mon.a on smithi002 2024-03-11T18:23:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:40 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/746626073' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: Reconfiguring daemon mon.a on smithi002 2024-03-11T18:23:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:40 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/746626073' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:40.661 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T18:23:40.662 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:23:40.662 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T18:23:40.763 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:23:40.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:40.886 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:23:40.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T18:23:40.926 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:23:40.926 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:41.001 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:23:41.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T18:23:41.037 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:23:41.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:23:41.107 INFO:tasks.cephadm:Adding mgr.x on smithi002 2024-03-11T18:23:41.107 INFO:tasks.cephadm:Adding mgr.y on smithi023 2024-03-11T18:23:41.107 INFO:tasks.cephadm:Adding mgr.z on smithi165 2024-03-11T18:23:41.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi002=x;smithi023=y;smithi165=z' 2024-03-11T18:23:41.382 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:23:41.675 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:41 smithi002 ceph-mon[31328]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:41.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:41 smithi165 ceph-mon[36634]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:41.904 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:41 smithi023 ceph-mon[36406]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:42.821 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 -- 172.21.15.165:0/3901408290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b50100050 msgr2=0x7f1b50102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:42.821 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 --2- 172.21.15.165:0/3901408290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b50100050 0x7f1b50102480 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009910 tx=0x7f1b48009c20 comp rx=0 tx=0).stop 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 -- 172.21.15.165:0/3901408290 shutdown_connections 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 --2- 172.21.15.165:0/3901408290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b50100050 0x7f1b50102480 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 -- 172.21.15.165:0/3901408290 >> 172.21.15.165:0/3901408290 conn(0x7f1b500fbc50 msgr2=0x7f1b500fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 -- 172.21.15.165:0/3901408290 shutdown_connections 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.817+0000 7f1b580d5700 1 -- 172.21.15.165:0/3901408290 wait complete. 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 Processor -- start 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 -- start start 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b50100050 0x7f1b5019d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b5019d8f0 0x7f1b501a2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b501a5690 con 0x7f1b5019d8f0 2024-03-11T18:23:42.822 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b501a5800 con 0x7f1b50100050 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b580d5700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b501a59c0 con 0x7f1b5019dd70 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b56672700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b56672700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:54410/0 (socket says 172.21.15.165:54410) 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b56672700 1 -- 172.21.15.165:0/693009424 learned_addr learned my addr 172.21.15.165:0/693009424 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.818+0000 7f1b55e71700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b50100050 0x7f1b5019d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 -- 172.21.15.165:0/693009424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b50100050 msgr2=0x7f1b5019d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b50100050 0x7f1b5019d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 -- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b5019d8f0 msgr2=0x7f1b501a2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b55670700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b5019d8f0 0x7f1b501a2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b5019d8f0 0x7f1b501a2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 -- 172.21.15.165:0/693009424 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b480095f0 con 0x7f1b5019dd70 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b56672700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000ba50 tx=0x7f1b4000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:42.823 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4000c9b0 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b501a5ca0 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b40014440 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b400075b0 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.819+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b501a6150 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.820+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f1b40007710 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.820+0000 7f1b4effd700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 0x7f1b3803b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.820+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b50197340 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.820+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1b4004b160 con 0x7f1b5019dd70 2024-03-11T18:23:42.824 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.821+0000 7f1b55e71700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 0x7f1b3803b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:42.825 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.822+0000 7f1b55e71700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 0x7f1b3803b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b48005230 tx=0x7f1b48005300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:42.826 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:42.825+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b4001b8b0 con 0x7f1b5019dd70 2024-03-11T18:23:42.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:42.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:42.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: Reconfiguring mon.b (monmap changed)... 2024-03-11T18:23:42.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:42.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:42.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:42.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:42 smithi165 ceph-mon[36634]: Reconfiguring daemon mon.b on smithi023 2024-03-11T18:23:43.033 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.031+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi023=y;smithi165=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b5002d8d0 con 0x7f1b380391e0 2024-03-11T18:23:43.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.045+0000 7f1b4effd700 1 -- 172.21.15.165:0/693009424 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1b5002d8d0 con 0x7f1b380391e0 2024-03-11T18:23:43.046 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 msgr2=0x7f1b3803b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 0x7f1b3803b6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1b48005230 tx=0x7f1b48005300 comp rx=0 tx=0).stop 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 msgr2=0x7f1b501a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000ba50 tx=0x7f1b4000bd60 comp rx=0 tx=0).stop 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 shutdown_connections 2024-03-11T18:23:43.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1b380391e0 0x7f1b3803b6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b5019dd70 0x7f1b501a4ef0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b50100050 0x7f1b5019d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 --2- 172.21.15.165:0/693009424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b5019d8f0 0x7f1b501a2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 >> 172.21.15.165:0/693009424 conn(0x7f1b500fbc50 msgr2=0x7f1b500fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 shutdown_connections 2024-03-11T18:23:43.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:23:43.047+0000 7f1b580d5700 1 -- 172.21.15.165:0/693009424 wait complete. 2024-03-11T18:23:43.154 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: Reconfiguring mon.b (monmap changed)... 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:42 smithi023 ceph-mon[36406]: Reconfiguring daemon mon.b on smithi023 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: Reconfiguring mon.b (monmap changed)... 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:42 smithi002 ceph-mon[31328]: Reconfiguring daemon mon.b on smithi023 2024-03-11T18:23:43.636 DEBUG:teuthology.orchestra.run.smithi023:mgr.y> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T18:23:43.639 DEBUG:teuthology.orchestra.run.smithi165:mgr.z> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T18:23:43.641 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T18:23:43.641 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:23:43.642 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T18:23:43.702 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T18:23:43.702 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-03-11T18:23:43.757 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 636 Links: 1 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 18:22:44.046620102 +0000 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 18:17:32.512616534 +0000 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 18:17:32.512616534 +0000 2024-03-11T18:23:43.758 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 18:17:32.512616534 +0000 2024-03-11T18:23:43.758 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T18:23:43.854 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T18:23:43.855 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T18:23:43.855 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000274584 s, 1.9 MB/s 2024-03-11T18:23:43.859 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T18:23:43.916 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-03-11T18:23:43.970 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 654 Links: 1 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 18:22:44.138618018 +0000 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 18:17:33.067603808 +0000 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 18:17:33.067603808 +0000 2024-03-11T18:23:43.971 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 18:17:33.067603808 +0000 2024-03-11T18:23:43.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T18:23:44.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:44.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.054 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.055 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:44.055 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:44.055 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:44.064 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T18:23:44.064 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T18:23:44.064 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000393349 s, 1.3 MB/s 2024-03-11T18:23:44.067 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:44.067 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.067 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.067 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.067 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:44.068 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:44.068 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:44.070 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T18:23:44.129 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-03-11T18:23:44.154 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T18:23:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:44.180 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T18:23:44.180 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 18:22:44.210616386 +0000 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 18:17:33.700589293 +0000 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 18:17:33.700589293 +0000 2024-03-11T18:23:44.181 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 18:17:33.700589293 +0000 2024-03-11T18:23:44.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T18:23:44.274 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T18:23:44.274 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T18:23:44.275 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000246384 s, 2.1 MB/s 2024-03-11T18:23:44.279 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T18:23:44.337 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 692 Links: 1 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 18:22:44.276614891 +0000 2024-03-11T18:23:44.391 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 18:17:34.276576085 +0000 2024-03-11T18:23:44.392 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 18:17:34.276576085 +0000 2024-03-11T18:23:44.392 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 18:17:34.276576085 +0000 2024-03-11T18:23:44.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T18:23:44.489 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-03-11T18:23:44.490 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-03-11T18:23:44.490 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000266997 s, 1.9 MB/s 2024-03-11T18:23:44.494 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T18:23:44.555 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-11T18:23:44.555 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T18:23:44.574 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T18:23:44.575 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-11T18:23:44.632 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T18:23:44.632 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:44.632 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-11T18:23:44.632 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:44.632 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:44.633 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 18:23:30.620516069 +0000 2024-03-11T18:23:44.633 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 18:16:30.994096580 +0000 2024-03-11T18:23:44.633 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 18:16:30.994096580 +0000 2024-03-11T18:23:44.633 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-11 18:16:30.994096580 +0000 2024-03-11T18:23:44.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T18:23:44.706 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T18:23:44.706 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T18:23:44.706 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000326747 s, 1.6 MB/s 2024-03-11T18:23:44.708 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T18:23:44.766 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-11T18:23:44.822 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T18:23:44.822 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 18:23:30.620516069 +0000 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 18:16:31.352093145 +0000 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 18:16:31.352093145 +0000 2024-03-11T18:23:44.823 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-11 18:16:31.352093145 +0000 2024-03-11T18:23:44.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T18:23:44.895 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T18:23:44.895 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T18:23:44.896 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000877335 s, 584 kB/s 2024-03-11T18:23:44.897 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T18:23:44.955 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-11T18:23:45.012 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T18:23:45.012 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.012 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-11T18:23:45.012 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.012 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.013 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 18:23:30.620516069 +0000 2024-03-11T18:23:45.013 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 18:16:31.715089663 +0000 2024-03-11T18:23:45.013 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 18:16:31.715089663 +0000 2024-03-11T18:23:45.013 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-11 18:16:31.715089663 +0000 2024-03-11T18:23:45.013 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T18:23:45.081 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:44 smithi023 ceph-mon[36406]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi023=y;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:45.081 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:44 smithi023 ceph-mon[36406]: Saving service mgr spec with placement smithi002=x;smithi023=y;smithi165=z;count:3 2024-03-11T18:23:45.081 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:44 smithi023 ceph-mon[36406]: Reconfiguring mon.c (monmap changed)... 2024-03-11T18:23:45.081 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:44 smithi023 ceph-mon[36406]: Reconfiguring daemon mon.c on smithi165 2024-03-11T18:23:45.083 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T18:23:45.083 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T18:23:45.083 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000284447 s, 1.8 MB/s 2024-03-11T18:23:45.085 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T18:23:45.141 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-11T18:23:45.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:44 smithi165 ceph-mon[36634]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi023=y;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:45.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:44 smithi165 ceph-mon[36634]: Saving service mgr spec with placement smithi002=x;smithi023=y;smithi165=z;count:3 2024-03-11T18:23:45.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:44 smithi165 ceph-mon[36634]: Reconfiguring mon.c (monmap changed)... 2024-03-11T18:23:45.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:44 smithi165 ceph-mon[36634]: Reconfiguring daemon mon.c on smithi165 2024-03-11T18:23:45.197 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T18:23:45.197 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.197 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-11T18:23:45.197 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.197 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.198 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-11 18:23:30.621516060 +0000 2024-03-11T18:23:45.198 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-11 18:16:32.078086181 +0000 2024-03-11T18:23:45.198 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-11 18:16:32.078086181 +0000 2024-03-11T18:23:45.198 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-11 18:16:32.078086181 +0000 2024-03-11T18:23:45.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T18:23:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:44 smithi002 ceph-mon[31328]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi002=x;smithi023=y;smithi165=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:44 smithi002 ceph-mon[31328]: Saving service mgr spec with placement smithi002=x;smithi023=y;smithi165=z;count:3 2024-03-11T18:23:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:44 smithi002 ceph-mon[31328]: Reconfiguring mon.c (monmap changed)... 2024-03-11T18:23:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:44 smithi002 ceph-mon[31328]: Reconfiguring daemon mon.c on smithi165 2024-03-11T18:23:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-11T18:23:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-11T18:23:45.267 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000291531 s, 1.8 MB/s 2024-03-11T18:23:45.268 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T18:23:45.324 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-03-11T18:23:45.324 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T18:23:45.340 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T18:23:45.340 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 18:23:20.839266337 +0000 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 18:16:08.344070124 +0000 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 18:16:08.344070124 +0000 2024-03-11T18:23:45.395 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 18:16:08.344070124 +0000 2024-03-11T18:23:45.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T18:23:45.462 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T18:23:45.463 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T18:23:45.463 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000196808 s, 2.6 MB/s 2024-03-11T18:23:45.464 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T18:23:45.520 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-03-11T18:23:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T18:23:45.575 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 18:23:20.840266321 +0000 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 18:16:08.679064500 +0000 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 18:16:08.679064500 +0000 2024-03-11T18:23:45.576 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 18:16:08.679064500 +0000 2024-03-11T18:23:45.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T18:23:45.641 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T18:23:45.641 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T18:23:45.641 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000178594 s, 2.9 MB/s 2024-03-11T18:23:45.642 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T18:23:45.699 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 18:23:20.840266321 +0000 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 18:16:09.021058759 +0000 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 18:16:09.021058759 +0000 2024-03-11T18:23:45.754 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 18:16:09.021058759 +0000 2024-03-11T18:23:45.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T18:23:45.820 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T18:23:45.820 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T18:23:45.820 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000170837 s, 3.0 MB/s 2024-03-11T18:23:45.821 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T18:23:45.877 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-03-11T18:23:45.931 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-03-11 18:23:20.840266321 +0000 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-03-11 18:16:09.374052834 +0000 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-03-11 18:16:09.374052834 +0000 2024-03-11T18:23:45.932 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-03-11 18:16:09.374052834 +0000 2024-03-11T18:23:45.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T18:23:45.997 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-03-11T18:23:45.998 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-03-11T18:23:45.998 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000213577 s, 2.4 MB/s 2024-03-11T18:23:45.999 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T18:23:46.054 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2024-03-11T18:23:46.054 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:45 smithi002 ceph-mon[31328]: Deploying daemon mgr.z on smithi165 2024-03-11T18:23:46.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:46.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:45 smithi165 ceph-mon[36634]: Deploying daemon mgr.z on smithi165 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:46.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:46.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:45 smithi023 ceph-mon[36406]: Deploying daemon mgr.z on smithi165 2024-03-11T18:23:46.508 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:46.905 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:46 smithi165 systemd[1]: Starting Ceph mgr.z for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 podman[38488]: 2024-03-11 18:23:46.908650566 +0000 UTC m=+0.018008042 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 podman[38488]: 2024-03-11 18:23:47.022022355 +0000 UTC m=+0.131379852 container create fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 podman[38488]: 2024-03-11 18:23:47.190898957 +0000 UTC m=+0.300256395 container init fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 podman[38488]: 2024-03-11 18:23:47.193424047 +0000 UTC m=+0.302781477 container start fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mgr[38507]: -- 172.21.15.165:0/387687699 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563dc3ac61a0 con 0x563dc3a9cc00 2024-03-11T18:23:47.266 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 bash[38488]: fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f 2024-03-11T18:23:47.654 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 systemd[1]: Started Ceph mgr.z for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:48.129 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:48.154 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:47.758+0000 7f3d3f9b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:23:48.154 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:47.851+0000 7f3d3f9b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:48.827 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:48.498+0000 7f3d3f9b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:23:48.827 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:48.590+0000 7f3d3f9b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:23:48.827 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:48.730+0000 7f3d3f9b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:23:49.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:48 smithi023 ceph-mon[36406]: Deploying daemon mgr.y on smithi023 2024-03-11T18:23:49.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-mon[36634]: Deploying daemon mgr.y on smithi023 2024-03-11T18:23:49.155 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:48.826+0000 7f3d3f9b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:23:49.155 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:48.910+0000 7f3d3f9b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:23:49.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:48 smithi002 ceph-mon[31328]: Deploying daemon mgr.y on smithi023 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0380411 s, 276 MB/s 2024-03-11T18:23:49.356 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-03-11T18:23:49.488 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:49.181+0000 7f3d3f9b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:23:49.488 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:49.297+0000 7f3d3f9b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:23:49.488 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:49.388+0000 7f3d3f9b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:23:49.489 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:49.486+0000 7f3d3f9b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:23:49.904 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:49.666+0000 7f3d3f9b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:23:50.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:49 smithi002 ceph-mon[31328]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:50.092 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2024-03-11T18:23:50.154 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:49 smithi023 ceph-mon[36406]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:50.216 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:49 smithi165 ceph-mon[36634]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:23:50.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:50.456+0000 7fc5fe074200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:23:50.975 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:50.976 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:50.976 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:50.976 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:50.976 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:51.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:50.699+0000 7f3d3f9b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:23:51.084 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:50.843+0000 7f3d3f9b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:23:51.243 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.155+0000 7fc5fe074200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:23:51.243 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.244 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.244 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.244 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:51.244 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:51.404 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:51.205+0000 7f3d3f9b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:23:51.404 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:51.296+0000 7f3d3f9b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:23:51.502 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.256+0000 7fc5fe074200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:23:51.502 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.402+0000 7fc5fe074200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:23:51.503 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.501+0000 7fc5fe074200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:23:51.870 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.593+0000 7fc5fe074200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:23:51.870 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.869+0000 7fc5fe074200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:23:51.904 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:51.493+0000 7f3d3f9b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:23:51.904 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:51.687+0000 7f3d3f9b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:23:52.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:52.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.130 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:52.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.154 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:51.982+0000 7fc5fe074200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:52.071+0000 7fc5fe074200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:52.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:52.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.234 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:51.974+0000 7f3d3f9b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:23:52.234 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:52.233+0000 7f3d3f9b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T18:23:52.235 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:52.654 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:52.469+0000 7f3d3f9b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:23:52.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:52.168+0000 7fc5fe074200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:23:52.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:52.350+0000 7fc5fe074200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:23:52.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 -- 172.21.15.2:0/2906488111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90103870 msgr2=0x7f2b9010a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2906488111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90103870 0x7f2b9010a120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b0d0 tx=0x7f2b8800b3e0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 -- 172.21.15.2:0/2906488111 shutdown_connections 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2906488111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90101cd0 0x7f2b901020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2906488111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90103870 0x7f2b9010a120 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2906488111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 0x7f2b90103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 -- 172.21.15.2:0/2906488111 >> 172.21.15.2:0/2906488111 conn(0x7f2b900fd270 msgr2=0x7f2b900ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 -- 172.21.15.2:0/2906488111 shutdown_connections 2024-03-11T18:23:52.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.691+0000 7f2b95a00700 1 -- 172.21.15.2:0/2906488111 wait complete. 2024-03-11T18:23:52.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.692+0000 7f2b95a00700 1 Processor -- start 2024-03-11T18:23:52.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 -- start start 2024-03-11T18:23:52.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 0x7f2b9019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90103870 0x7f2b901a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901a1b20 con 0x7f2b90102ed0 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901a1c60 con 0x7f2b90101cd0 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b95a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b901a1df0 con 0x7f2b90103870 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b8ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b8ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43302/0 (socket says 172.21.15.2:43302) 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b8e000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 0x7f2b9019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b8ea00700 1 -- 172.21.15.2:0/2872721727 learned_addr learned my addr 172.21.15.2:0/2872721727 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:23:52.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.693+0000 7f2b8f400700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90103870 0x7f2b901a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:52.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 -- 172.21.15.2:0/2872721727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90103870 msgr2=0x7f2b901a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90103870 0x7f2b901a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 -- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 msgr2=0x7f2b9019a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 0x7f2b9019a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 -- 172.21.15.2:0/2872721727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88009d00 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b8ea00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e970 tx=0x7f2b8000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80009790 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b8000ccc0 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b80003d60 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.694+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b901a20d0 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.695+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b901a2580 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.696+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45824+0+0 (secure 0 0 0) 0x7f2b8000c7e0 con 0x7f2b90101cd0 2024-03-11T18:23:52.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.696+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b90068930 con 0x7f2b90101cd0 2024-03-11T18:23:52.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.696+0000 7f2b87400700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 0x7f2b7403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:23:52.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.696+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b80014070 con 0x7f2b90101cd0 2024-03-11T18:23:52.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.697+0000 7f2b8e000700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 0x7f2b7403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:23:52.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.697+0000 7f2b8e000700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 0x7f2b7403b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009c60 tx=0x7f2b7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:23:52.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.707+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b80004220 con 0x7f2b90101cd0 2024-03-11T18:23:52.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:52.955+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b900755c0 con 0x7f2b74038f90 2024-03-11T18:23:53.101 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:52 smithi002 ceph-mon[31328]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T18:23:53.101 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:52 smithi002 ceph-mon[31328]: Reconfiguring daemon mgr.x on smithi002 2024-03-11T18:23:53.101 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:23:53.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:23:53.102 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:53.268 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-mon[36634]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T18:23:53.268 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-mon[36634]: Reconfiguring daemon mgr.x on smithi002 2024-03-11T18:23:53.268 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:23:53.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:23:53.269 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:53.377 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-mon[36406]: Reconfiguring mgr.x (unknown last config time)... 2024-03-11T18:23:53.377 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-mon[36406]: Reconfiguring daemon mgr.x on smithi002 2024-03-11T18:23:53.377 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:23:53.378 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:23:53.378 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:53.654 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:53.267+0000 7f3d3f9b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:23:53.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:53.376+0000 7fc5fe074200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:23:53.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:53.511+0000 7fc5fe074200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:23:54.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:53 smithi002 ceph-mon[31328]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:54.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:53 smithi002 ceph-mon[31328]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:54.120 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.121 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.143 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-mon[36634]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-mon[36634]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:53.833+0000 7f3d3f9b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:53.928+0000 7f3d3f9b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:23:54.144 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:23:54.142+0000 7f3d3f9b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-mon[36406]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-mon[36406]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:53.853+0000 7fc5fe074200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:53.945+0000 7fc5fe074200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:23:54.148 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:54.147+0000 7fc5fe074200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:23:54.404 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:54.338+0000 7fc5fe074200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:23:54.897 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:54.628+0000 7fc5fe074200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:23:54.897 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:54.896+0000 7fc5fe074200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:23:55.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:55.039+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 14) v1 ==== 91023+0+0 (secure 0 0 0) 0x7f2b800187b0 con 0x7f2b90101cd0 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: Standby manager daemon z started 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:23:55.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:54 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:55.154 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:55 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:55.139+0000 7fc5fe074200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:23:55.154 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: Standby manager daemon z started 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:23:55.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:54 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: Standby manager daemon z started 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:23:55.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:54 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/2534705302' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:56.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:55 smithi002 ceph-mon[31328]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:56.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:55 smithi002 ceph-mon[31328]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-11T18:23:56.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:55 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:23:56.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:55 smithi165 ceph-mon[36634]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:55 smithi165 ceph-mon[36634]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-11T18:23:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:55 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:23:56.405 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:55 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:55.965+0000 7fc5fe074200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:23:56.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:55 smithi023 ceph-mon[36406]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:55 smithi023 ceph-mon[36406]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-11T18:23:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:55 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:23:56.876 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:56 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:56.554+0000 7fc5fe074200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:23:56.876 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:56 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:56.654+0000 7fc5fe074200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:23:56.876 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:23:56 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:23:56.875+0000 7fc5fe074200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:23:57.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:23:57.008+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mon.2 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 15) v1 ==== 136222+0+0 (secure 0 0 0) 0x7f2b8001dd50 con 0x7f2b90101cd0 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: Standby manager daemon y started 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1574836513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]': finished 2024-03-11T18:23:57.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T18:23:57.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: Standby manager daemon y started 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1574836513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]': finished 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: Standby manager daemon y started 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:23:57.406 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/294963864' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1574836513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]: dispatch 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a773fb9c-07bb-4b3e-94d4-8c382a21561e"}]': finished 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T18:23:57.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:23:58.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:58 smithi165 ceph-mon[36634]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:58 smithi165 ceph-mon[36634]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:58 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2990197409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:58 smithi023 ceph-mon[36406]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:58 smithi023 ceph-mon[36406]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:23:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:58 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2990197409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:23:58.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:58 smithi002 ceph-mon[31328]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:58.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:58 smithi002 ceph-mon[31328]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-11T18:23:58.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:23:58.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:58 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2990197409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:23:59.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:23:59 smithi002 ceph-mon[31328]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:59.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:23:59 smithi023 ceph-mon[36406]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:23:59.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:23:59 smithi165 ceph-mon[36634]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:01.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:01 smithi165 ceph-mon[36634]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:01.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:01 smithi023 ceph-mon[36406]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:01.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:01 smithi002 ceph-mon[31328]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:03.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:03 smithi165 ceph-mon[36634]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:03.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:03 smithi023 ceph-mon[36406]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:03.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:03 smithi002 ceph-mon[31328]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:05.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:05 smithi165 ceph-mon[36634]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:05.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:05 smithi023 ceph-mon[36406]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:05.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:05 smithi002 ceph-mon[31328]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:06.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T18:24:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:06 smithi002 ceph-mon[31328]: Deploying daemon osd.0 on smithi002 2024-03-11T18:24:06.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T18:24:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:06 smithi165 ceph-mon[36634]: Deploying daemon osd.0 on smithi002 2024-03-11T18:24:06.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T18:24:06.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:06.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:06 smithi023 ceph-mon[36406]: Deploying daemon osd.0 on smithi002 2024-03-11T18:24:07.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:07 smithi002 ceph-mon[31328]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:07.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:07 smithi165 ceph-mon[36634]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:07.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:07 smithi023 ceph-mon[36406]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:09.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:09 smithi002 ceph-mon[31328]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:09.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:09 smithi165 ceph-mon[36634]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:09.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:09 smithi023 ceph-mon[36406]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:11.250 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:11 smithi002 ceph-mon[31328]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:11.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:11 smithi165 ceph-mon[36634]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:11.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:11 smithi023 ceph-mon[36406]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:13.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:13 smithi002 ceph-mon[31328]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:13.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:13 smithi165 ceph-mon[36634]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:13.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:13 smithi023 ceph-mon[36406]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:15.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:15.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:15.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:15.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:16.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:15 smithi165 ceph-mon[36634]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:16.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:15 smithi023 ceph-mon[36406]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:16.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:15 smithi002 ceph-mon[31328]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:17.950 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:17.951 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:17.951 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:17.951 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:17.951 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:17.951 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:18.549 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2024-03-11T18:24:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.547+0000 7f2b87400700 1 -- 172.21.15.2:0/2872721727 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2b900755c0 con 0x7f2b74038f90 2024-03-11T18:24:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 msgr2=0x7f2b7403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 0x7f2b7403b450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009c60 tx=0x7f2b7c009260 comp rx=0 tx=0).stop 2024-03-11T18:24:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 msgr2=0x7f2b90199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e970 tx=0x7f2b8000ec80 comp rx=0 tx=0).stop 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 shutdown_connections 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b74038f90 0x7f2b7403b450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b90103870 0x7f2b901a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b90101cd0 0x7f2b90199f90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 --2- 172.21.15.2:0/2872721727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b90102ed0 0x7f2b9019a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 >> 172.21.15.2:0/2872721727 conn(0x7f2b900fd270 msgr2=0x7f2b90073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 shutdown_connections 2024-03-11T18:24:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:24:18.551+0000 7f2b95a00700 1 -- 172.21.15.2:0/2872721727 wait complete. 2024-03-11T18:24:19.072 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:18 smithi023 ceph-mon[36406]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T18:24:19.072 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.072 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.072 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:19.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:18 smithi165 ceph-mon[36634]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T18:24:19.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:19.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:18 smithi002 ceph-mon[31328]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T18:24:19.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:19.244 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:19.300 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T18:24:19.305 INFO:tasks.cephadm:Deploying osd.1 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-11T18:24:19.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T18:24:19.559 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:24:19.935 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:19 smithi023 ceph-mon[36406]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:19.935 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:19 smithi023 ceph-mon[36406]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T18:24:19.936 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:19 smithi023 ceph-mon[36406]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T18:24:19.936 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:19.936 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:19 smithi023 ceph-mon[36406]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:24:20.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:19 smithi165 ceph-mon[36634]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:20.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:19 smithi165 ceph-mon[36634]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T18:24:20.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:19 smithi165 ceph-mon[36634]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T18:24:20.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:20.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:19 smithi165 ceph-mon[36634]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:24:20.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-mon[31328]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:20.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-mon[31328]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T18:24:20.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-mon[31328]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T18:24:20.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:20.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-mon[31328]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:24:20.253 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0[46512]: 2024-03-11T18:24:19.862+0000 7ff3a1200700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T18:24:20.253 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:24:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0[46512]: 2024-03-11T18:24:19.869+0000 7ff39c000700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:24:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:20 smithi023 ceph-mon[36406]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:24:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:20 smithi023 ceph-mon[36406]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T18:24:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:20.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:20 smithi002 ceph-mon[31328]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:24:20.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:20 smithi002 ceph-mon[31328]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T18:24:20.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:20.959 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:21.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:20 smithi165 ceph-mon[36634]: from='osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:24:21.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:20 smithi165 ceph-mon[36634]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T18:24:21.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:21.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:21.519 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T18:24:21.519 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T18:24:21.519 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T18:24:21.519 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T18:24:21.520 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00753203 s, 1.4 GB/s 2024-03-11T18:24:21.520 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] boot 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T18:24:22.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:21 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:22.138 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-11T18:24:22.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:24:22.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:24:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] boot 2024-03-11T18:24:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T18:24:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:21 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:22.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:24:22.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:24:22.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T18:24:22.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: osd.0 [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] boot 2024-03-11T18:24:22.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T18:24:22.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:21 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:24:22.384 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:24:23.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:22 smithi165 ceph-mon[36634]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T18:24:23.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:22 smithi023 ceph-mon[36406]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T18:24:23.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:22 smithi002 ceph-mon[31328]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 -- 172.21.15.23:0/3119500633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c102ee0 msgr2=0x7f748c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 --2- 172.21.15.23:0/3119500633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c102ee0 0x7f748c103340 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f747c0099c0 tx=0x7f747c009cd0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 -- 172.21.15.23:0/3119500633 shutdown_connections 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 --2- 172.21.15.23:0/3119500633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c103880 0x7f748c10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 --2- 172.21.15.23:0/3119500633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f748c101ce0 0x7f748c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 --2- 172.21.15.23:0/3119500633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c102ee0 0x7f748c103340 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 -- 172.21.15.23:0/3119500633 >> 172.21.15.23:0/3119500633 conn(0x7f748c0fd2c0 msgr2=0x7f748c0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:24:23.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 -- 172.21.15.23:0/3119500633 shutdown_connections 2024-03-11T18:24:23.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.879+0000 7f7493c5f700 1 -- 172.21.15.23:0/3119500633 wait complete. 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 Processor -- start 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 -- start start 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f748c101ce0 0x7f748c197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c102ee0 0x7f748c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c19f850 con 0x7f748c101ce0 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c19f990 con 0x7f748c103880 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f7493c5f700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c19fb50 con 0x7f748c102ee0 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:23.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53278/0 (socket says 172.21.15.23:53278) 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 -- 172.21.15.23:0/2903275852 learned_addr learned my addr 172.21.15.23:0/2903275852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74911fa700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c102ee0 0x7f748c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 -- 172.21.15.23:0/2903275852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c102ee0 msgr2=0x7f748c198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c102ee0 0x7f748c198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 -- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f748c101ce0 msgr2=0x7f748c197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f748c101ce0 0x7f748c197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.880+0000 7f74921fc700 1 -- 172.21.15.23:0/2903275852 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0095f0 con 0x7f748c103880 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f74921fc700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f748400b970 tx=0x7f748400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748400c930 con 0x7f748c103880 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c19fe30 con 0x7f748c103880 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7484014440 con 0x7f748c103880 2024-03-11T18:24:23.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74840075d0 con 0x7f748c103880 2024-03-11T18:24:23.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.881+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c1a02d0 con 0x7f748c103880 2024-03-11T18:24:23.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.882+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c109040 con 0x7f748c103880 2024-03-11T18:24:23.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.883+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 136222+0+0 (secure 0 0 0) 0x7f74840145b0 con 0x7f748c103880 2024-03-11T18:24:23.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.884+0000 7f7482ffd700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 0x7f74780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:23.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.884+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f7484006e40 con 0x7f748c103880 2024-03-11T18:24:23.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.884+0000 7f74919fb700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 0x7f74780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:23.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.885+0000 7f74919fb700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 0x7f74780a3ef0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f748c1991e0 tx=0x7f7488006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:24:23.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:23.888+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7484095c80 con 0x7f748c103880 2024-03-11T18:24:24.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:23 smithi002 ceph-mon[31328]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:24.098+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f748c02d920 con 0x7f74780a1a30 2024-03-11T18:24:24.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:23 smithi165 ceph-mon[36634]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:24.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:23 smithi023 ceph-mon[36406]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:25.106 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:25.106 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:25.106 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.106 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.106 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.107 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:25.107 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.107 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.107 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:25.107 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:25.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:25.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:25.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:25.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:25.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:26.016 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:25 smithi023 ceph-mon[36406]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:26.017 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:25 smithi023 ceph-mon[36406]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:26.017 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:25 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi002 2024-03-11T18:24:26.017 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:25 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 17095M 2024-03-11T18:24:26.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:25 smithi165 ceph-mon[36634]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:25 smithi165 ceph-mon[36634]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:25 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi002 2024-03-11T18:24:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:25 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 17095M 2024-03-11T18:24:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:25 smithi002 ceph-mon[31328]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:25 smithi002 ceph-mon[31328]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:25 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi002 2024-03-11T18:24:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:25 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 17095M 2024-03-11T18:24:27.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:26 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/1372043999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:26 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:26 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]': finished 2024-03-11T18:24:27.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:26 smithi002 ceph-mon[31328]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T18:24:27.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:26 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:26 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/1372043999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:26 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:26 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]': finished 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:26 smithi023 ceph-mon[36406]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:26 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:26 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/1372043999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:26 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]: dispatch 2024-03-11T18:24:27.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:26 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec6ef905-1b64-44d7-bbd0-d52bc91b22df"}]': finished 2024-03-11T18:24:27.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:26 smithi165 ceph-mon[36634]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T18:24:27.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:26 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:28.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:27 smithi165 ceph-mon[36634]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:27 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/1923317482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:28.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:27 smithi023 ceph-mon[36406]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:28.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:27 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/1923317482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:28.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:27 smithi002 ceph-mon[31328]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:28.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:27 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/1923317482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:30.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:29 smithi165 ceph-mon[36634]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:29 smithi023 ceph-mon[36406]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:29 smithi002 ceph-mon[31328]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:32.111 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:31 smithi023 ceph-mon[36406]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:32.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:31 smithi165 ceph-mon[36634]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:31 smithi002 ceph-mon[31328]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:34.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:33 smithi023 ceph-mon[36406]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:34.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:33 smithi002 ceph-mon[31328]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:34.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:33 smithi165 ceph-mon[36634]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:35.132 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T18:24:35.133 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:35.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T18:24:35.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:35.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T18:24:35.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:36.094 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:35 smithi023 ceph-mon[36406]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:36.094 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:35 smithi023 ceph-mon[36406]: Deploying daemon osd.1 on smithi023 2024-03-11T18:24:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:35 smithi002 ceph-mon[31328]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:35 smithi002 ceph-mon[31328]: Deploying daemon osd.1 on smithi023 2024-03-11T18:24:36.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:35 smithi165 ceph-mon[36634]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:36.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:35 smithi165 ceph-mon[36634]: Deploying daemon osd.1 on smithi023 2024-03-11T18:24:38.236 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:37 smithi023 ceph-mon[36406]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:38.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:37 smithi002 ceph-mon[31328]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:38.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:37 smithi165 ceph-mon[36634]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:39 smithi023 ceph-mon[36406]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:40.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:39 smithi002 ceph-mon[31328]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:40.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:39 smithi165 ceph-mon[36634]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:41.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:42.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:41 smithi023 ceph-mon[36406]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:41 smithi002 ceph-mon[31328]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:42.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:41 smithi165 ceph-mon[36634]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:43.091 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 1 on host 'smithi023' 2024-03-11T18:24:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.088+0000 7f7482ffd700 1 -- 172.21.15.23:0/2903275852 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f748c02d920 con 0x7f74780a1a30 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 msgr2=0x7f74780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 0x7f74780a3ef0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f748c1991e0 tx=0x7f7488006d20 comp rx=0 tx=0).stop 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 msgr2=0x7f748c19f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f748400b970 tx=0x7f748400bc80 comp rx=0 tx=0).stop 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 shutdown_connections 2024-03-11T18:24:43.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f748c103880 0x7f748c19f280 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f748c101ce0 0x7f748c197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f74780a1a30 0x7f74780a3ef0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 --2- 172.21.15.23:0/2903275852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f748c102ee0 0x7f748c198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 >> 172.21.15.23:0/2903275852 conn(0x7f748c0fd2c0 msgr2=0x7f748c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 shutdown_connections 2024-03-11T18:24:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:24:43.091+0000 7f7493c5f700 1 -- 172.21.15.23:0/2903275852 wait complete. 2024-03-11T18:24:44.060 DEBUG:teuthology.orchestra.run.smithi023:osd.1> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T18:24:44.063 INFO:tasks.cephadm:Deploying osd.2 on smithi165 with /dev/vg_nvme/lv_4... 2024-03-11T18:24:44.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T18:24:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:43 smithi002 ceph-mon[31328]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:43 smithi002 ceph-mon[31328]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:43 smithi002 ceph-mon[31328]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:44.308 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:43 smithi023 ceph-mon[36406]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:43 smithi023 ceph-mon[36406]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:43 smithi023 ceph-mon[36406]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:43 smithi165 ceph-mon[36634]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:43 smithi165 ceph-mon[36634]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:43 smithi165 ceph-mon[36634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T18:24:45.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T18:24:45.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T18:24:45.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:45.327 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:45.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:45.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T18:24:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T18:24:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:24:45.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:45.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:45.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:46.062 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1[40995]: 2024-03-11T18:24:45.754+0000 7f561b505700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T18:24:46.062 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:24:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1[40995]: 2024-03-11T18:24:45.758+0000 7f56132f0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:24:46.116 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T18:24:46.116 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T18:24:46.117 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-11T18:24:46.117 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-11T18:24:46.117 INFO:teuthology.orchestra.run.smithi165.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113692 s, 922 MB/s 2024-03-11T18:24:46.117 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-11T18:24:46.170 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:46 smithi165 ceph-mon[36634]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:24:46.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:46 smithi165 ceph-mon[36634]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T18:24:46.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:46 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:46 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.171 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:46 smithi165 ceph-mon[36634]: from='osd.1 ' entity='osd.1' 2024-03-11T18:24:46.333 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:46 smithi023 ceph-mon[36406]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:24:46.333 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:46 smithi023 ceph-mon[36406]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T18:24:46.333 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:46 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.333 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:46 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.333 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:46 smithi023 ceph-mon[36406]: from='osd.1 ' entity='osd.1' 2024-03-11T18:24:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:46 smithi002 ceph-mon[31328]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:24:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:46 smithi002 ceph-mon[31328]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T18:24:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:46 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:46 smithi002 ceph-mon[31328]: from='osd.1 ' entity='osd.1' 2024-03-11T18:24:46.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-03-11T18:24:46.980 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] boot 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T18:24:47.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:47 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] boot 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T18:24:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:47 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: osd.1 [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] boot 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T18:24:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:47 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:24:48.412 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 -- 172.21.15.165:0/845578070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc102ed0 msgr2=0x7f81bc103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/845578070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc102ed0 0x7f81bc103330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009990 tx=0x7f81ac009ca0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 -- 172.21.15.165:0/845578070 shutdown_connections 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/845578070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81bc103870 0x7f81bc10a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/845578070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 0x7f81bc1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/845578070 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc102ed0 0x7f81bc103330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 -- 172.21.15.165:0/845578070 >> 172.21.15.165:0/845578070 conn(0x7f81bc0fd270 msgr2=0x7f81bc0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 -- 172.21.15.165:0/845578070 shutdown_connections 2024-03-11T18:24:48.413 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.411+0000 7f81c2e3b700 1 -- 172.21.15.165:0/845578070 wait complete. 2024-03-11T18:24:48.415 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 Processor -- start 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 -- start start 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 0x7f81bc197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81bc102ed0 0x7f81bc1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19f930 con 0x7f81bc102ed0 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19fa70 con 0x7f81bc101cd0 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c2e3b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc19fc00 con 0x7f81bc103870 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56970/0 (socket says 172.21.15.165:56970) 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 -- 172.21.15.165:0/654293978 learned_addr learned my addr 172.21.15.165:0/654293978 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c0bd7700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 0x7f81bc197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:48.416 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 -- 172.21.15.165:0/654293978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 msgr2=0x7f81bc197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 0x7f81bc197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 -- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81bc102ed0 msgr2=0x7f81bc1982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81bc102ed0 0x7f81bc1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.412+0000 7f81c13d8700 1 -- 172.21.15.165:0/654293978 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac0095f0 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81c13d8700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f81b400b580 tx=0x7f81b400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4011830 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc19fee0 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81b4011e70 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b400f2e0 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.413+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc1a03d0 con 0x7f81bc103870 2024-03-11T18:24:48.417 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.414+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc109090 con 0x7f81bc103870 2024-03-11T18:24:48.418 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.415+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 15) v1 ==== 136222+0+0 (secure 0 0 0) 0x7f81b4011990 con 0x7f81bc103870 2024-03-11T18:24:48.418 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.415+0000 7f81b1ffb700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 0x7f81a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:24:48.418 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.416+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f81b40cb2c0 con 0x7f81bc103870 2024-03-11T18:24:48.419 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.416+0000 7f81c0bd7700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 0x7f81a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:24:48.419 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.417+0000 7f81c0bd7700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 0x7f81a40a3ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f81bc102d30 tx=0x7f81b80094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:24:48.421 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.420+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f81b4095d20 con 0x7f81bc103870 2024-03-11T18:24:48.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:24:48.626+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f81bc02d8e0 con 0x7f81a40a1a30 2024-03-11T18:24:48.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T18:24:48.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi023 2024-03-11T18:24:48.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:48.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:48.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:48.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 17080M 2024-03-11T18:24:48.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:48.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:48.908 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:48.908 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:48.908 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:48.908 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:48 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.056 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T18:24:49.057 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi023 2024-03-11T18:24:49.057 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.057 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.057 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:49.057 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 17080M 2024-03-11T18:24:49.058 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.058 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.058 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:49.058 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.058 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:49.059 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:48 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi023 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 17080M 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:24:49.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:48 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:49 smithi002 ceph-mon[31328]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:49 smithi002 ceph-mon[31328]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:50.261 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:49 smithi165 ceph-mon[36634]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:50.262 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:49 smithi165 ceph-mon[36634]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:50.347 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:49 smithi023 ceph-mon[36406]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:50.348 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:49 smithi023 ceph-mon[36406]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/705320963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]': finished 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T18:24:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:24:52.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:51 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/1294843811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:52.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/705320963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]': finished 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:51 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/1294843811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/705320963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]: dispatch 2024-03-11T18:24:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa7a2e4d-c63a-479a-b306-c3c983dea728"}]': finished 2024-03-11T18:24:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T18:24:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:24:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:51 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/1294843811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:24:54.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:53 smithi002 ceph-mon[31328]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:54.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:53 smithi165 ceph-mon[36634]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:54.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:53 smithi023 ceph-mon[36406]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:56.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:55 smithi002 ceph-mon[31328]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:56.257 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:55 smithi165 ceph-mon[36634]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:56.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:55 smithi023 ceph-mon[36406]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:58.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:57 smithi002 ceph-mon[31328]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:58.291 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:57 smithi165 ceph-mon[36634]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:58.404 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:57 smithi023 ceph-mon[36406]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:24:59.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T18:24:59.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:59.252 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T18:24:59.252 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:24:59.404 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T18:24:59.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:59 smithi002 ceph-mon[31328]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:24:59 smithi002 ceph-mon[31328]: Deploying daemon osd.2 on smithi165 2024-03-11T18:25:00.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:59 smithi165 ceph-mon[36634]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:00.328 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:24:59 smithi165 ceph-mon[36634]: Deploying daemon osd.2 on smithi165 2024-03-11T18:25:00.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:59 smithi023 ceph-mon[36406]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:00.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:24:59 smithi023 ceph-mon[36406]: Deploying daemon osd.2 on smithi165 2024-03-11T18:25:02.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:01 smithi002 ceph-mon[31328]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:02.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:01 smithi165 ceph-mon[36634]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:02.404 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:01 smithi023 ceph-mon[36406]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:04.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:04 smithi165 ceph-mon[36634]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:04.087 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.087 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.088 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:04 smithi002 ceph-mon[31328]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:04.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:04 smithi023 ceph-mon[36406]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:04.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:04.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:06.355 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:06 smithi165 ceph-mon[36634]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:06 smithi023 ceph-mon[36406]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:06 smithi002 ceph-mon[31328]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:06.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.657+0000 7f81b1ffb700 1 -- 172.21.15.165:0/654293978 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f81bc02d8e0 con 0x7f81a40a1a30 2024-03-11T18:25:06.659 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 2 on host 'smithi165' 2024-03-11T18:25:06.661 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 msgr2=0x7f81a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 0x7f81a40a3ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f81bc102d30 tx=0x7f81b80094f0 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 msgr2=0x7f81bc19f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f81b400b580 tx=0x7f81b400b940 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 shutdown_connections 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f81a40a1a30 0x7f81a40a3ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81bc103870 0x7f81bc19f360 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81bc101cd0 0x7f81bc197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 --2- 172.21.15.165:0/654293978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81bc102ed0 0x7f81bc1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 >> 172.21.15.165:0/654293978 conn(0x7f81bc0fd270 msgr2=0x7f81bc1061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 shutdown_connections 2024-03-11T18:25:06.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:25:06.660+0000 7f81c2e3b700 1 -- 172.21.15.165:0/654293978 wait complete. 2024-03-11T18:25:07.230 DEBUG:teuthology.orchestra.run.smithi165:osd.2> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T18:25:07.233 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_3... 2024-03-11T18:25:07.233 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T18:25:07.584 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:25:07.358+0000 7f573c2a27c0 -1 osd.2 0 log_to_monitors true 2024-03-11T18:25:07.712 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:25:07.852 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:07.852 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.852 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.853 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.853 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.853 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:07.853 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:07 smithi165 ceph-mon[36634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:07 smithi023 ceph-mon[36406]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:07.906 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:07 smithi002 ceph-mon[31328]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.094 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:09 smithi002 ceph-mon[31328]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-mon[36634]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: from='osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:25:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:09 smithi023 ceph-mon[36406]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:09.905 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:25:09.763+0000 7f5731907700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T18:25:09.905 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:25:09.768+0000 7f57296f2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:25:10.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:10 smithi002 ceph-mon[31328]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:25:10.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:10 smithi002 ceph-mon[31328]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T18:25:10.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:10 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:10.148 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:10 smithi002 ceph-mon[31328]: from='osd.2 ' entity='osd.2' 2024-03-11T18:25:10.404 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:10 smithi023 ceph-mon[36406]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:10 smithi023 ceph-mon[36406]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:10 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:10 smithi023 ceph-mon[36406]: from='osd.2 ' entity='osd.2' 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:10 smithi165 ceph-mon[36634]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:10 smithi165 ceph-mon[36634]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T18:25:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:10 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:10.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:10 smithi165 ceph-mon[36634]: from='osd.2 ' entity='osd.2' 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0377185 s, 278 MB/s 2024-03-11T18:25:10.734 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi165 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 17080M 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] boot 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T18:25:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi165 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 17080M 2024-03-11T18:25:11.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] boot 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T18:25:11.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:25:11.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:25:11.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi165 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 17080M 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:11.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:11.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: osd.2 [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] boot 2024-03-11T18:25:11.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T18:25:11.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:25:11.554 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2024-03-11T18:25:12.017 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:25:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:13 smithi002 ceph-mon[31328]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T18:25:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:13 smithi002 ceph-mon[31328]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:13 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:13.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:13 smithi165 ceph-mon[36634]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T18:25:13.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:13 smithi165 ceph-mon[36634]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:13.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:13 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:13.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:13 smithi023 ceph-mon[36406]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T18:25:13.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:13 smithi023 ceph-mon[36406]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:13.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:13 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:14.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 -- 172.21.15.2:0/2116780265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 msgr2=0x7f4c4010c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:14.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 --2- 172.21.15.2:0/2116780265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c4010c060 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009f90 tx=0x7f4c3800b330 comp rx=0 tx=0).stop 2024-03-11T18:25:14.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 -- 172.21.15.2:0/2116780265 shutdown_connections 2024-03-11T18:25:14.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 --2- 172.21.15.2:0/2116780265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c4010c060 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:14.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 --2- 172.21.15.2:0/2116780265 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 0x7f4c400740a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:14.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 --2- 172.21.15.2:0/2116780265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:14.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 -- 172.21.15.2:0/2116780265 >> 172.21.15.2:0/2116780265 conn(0x7f4c400fd290 msgr2=0x7f4c400ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:25:14.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.127+0000 7f4c45800700 1 -- 172.21.15.2:0/2116780265 shutdown_connections 2024-03-11T18:25:14.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.128+0000 7f4c45800700 1 -- 172.21.15.2:0/2116780265 wait complete. 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 Processor -- start 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 -- start start 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 0x7f4c4019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c401a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401a1aa0 con 0x7f4c40075c40 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401a1be0 con 0x7f4c40074670 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.129+0000 7f4c45800700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401a1d70 con 0x7f4c40109bd0 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:14.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60200/0 (socket says 172.21.15.2:60200) 2024-03-11T18:25:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 -- 172.21.15.2:0/1278525003 learned_addr learned my addr 172.21.15.2:0/1278525003 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:25:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3f400700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c401a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3e000700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 0x7f4c4019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:14.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 -- 172.21.15.2:0/1278525003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 msgr2=0x7f4c401a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c401a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 -- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 msgr2=0x7f4c4019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 0x7f4c4019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 -- 172.21.15.2:0/1278525003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0095f0 con 0x7f4c40074670 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.130+0000 7f4c3ea00700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e9a0 tx=0x7f4c3000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.131+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30009790 con 0x7f4c40074670 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.131+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c38009c40 con 0x7f4c40074670 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.131+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c3000cbd0 con 0x7f4c40074670 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.131+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30018680 con 0x7f4c40074670 2024-03-11T18:25:14.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.131+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c401a22d0 con 0x7f4c40074670 2024-03-11T18:25:14.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.132+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c40068930 con 0x7f4c40074670 2024-03-11T18:25:14.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.134+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 136222+0+0 (secure 0 0 0) 0x7f4c3000cd40 con 0x7f4c40074670 2024-03-11T18:25:14.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.135+0000 7f4c37400700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 0x7f4c240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.136+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f4c30014070 con 0x7f4c40074670 2024-03-11T18:25:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.136+0000 7f4c3e000700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 0x7f4c240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:14.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.137+0000 7f4c3e000700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 0x7f4c240a3ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005fd0 tx=0x7f4c2c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:25:14.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.140+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c30052d20 con 0x7f4c40074670 2024-03-11T18:25:14.251 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:14.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:14 smithi002 ceph-mon[31328]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T18:25:14.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:14.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:14.383+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c4002d9b0 con 0x7f4c240a1a30 2024-03-11T18:25:14.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:14.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:14 smithi165 ceph-mon[36634]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T18:25:14.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:14 smithi023 ceph-mon[36406]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T18:25:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:25:15.064 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:14 smithi002 sudo[51277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:15.064 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:14 smithi002 sudo[51277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:15.064 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:14 smithi002 sudo[51277]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:15.064 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:14 smithi002 sudo[51277]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:15.365 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:15.366 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:15.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:15.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T18:25:15.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:15.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:15.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:15.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:15.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:15 smithi002 sudo[51332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:15.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:15 smithi002 sudo[51332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:15.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:15 smithi002 sudo[51332]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:15.633 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:25:15 smithi002 sudo[51332]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:16.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:16 smithi023 ceph-mon[36406]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T18:25:16.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:16 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:16.156 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:15 smithi023 sudo[45154]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:16.156 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:15 smithi023 sudo[45154]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:16.156 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:15 smithi023 sudo[45154]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:16.156 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:15 smithi023 sudo[45154]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:16.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:16 smithi165 ceph-mon[36634]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T18:25:16.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:16 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:16.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:16 smithi002 ceph-mon[31328]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T18:25:16.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:16 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:16.655 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:16 smithi023 sudo[45157]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:16.655 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:16 smithi023 sudo[45157]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:16.655 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:16 smithi023 sudo[45157]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:16.655 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:25:16 smithi023 sudo[45157]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:16.863 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45434]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:16.864 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45434]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:16.864 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45434]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:16.864 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45434]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:17.154 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45437]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:17.154 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45437]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:17.155 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45437]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:17.155 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:25:16 smithi165 sudo[45437]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:17 smithi002 sudo[51386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:17 smithi002 sudo[51386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:17 smithi002 sudo[51386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:17 smithi002 sudo[51386]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:18.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:18.098+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f4c30096bf0 con 0x7f4c40074670 2024-03-11T18:25:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:18 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 sudo[51420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 sudo[51420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 sudo[51420]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 sudo[51420]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:18 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:18.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45440]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:18.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45440]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:18.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45440]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:18.584 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45440]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45443]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45443]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45443]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 sudo[45443]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:18.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:18 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:19.306 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T18:25:19.306 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:19.306 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:19.306 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45160]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45163]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T18:25:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45163]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T18:25:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45163]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T18:25:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 sudo[45163]: pam_unix(sudo:session): session closed for user root 2024-03-11T18:25:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3744898169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]': finished 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:19 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3744898169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]': finished 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:19 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3744898169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332dc7bc-07d1-4637-b31b-8c2e2171d1da"}]': finished 2024-03-11T18:25:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T18:25:20.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:25:20.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:25:20.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:19 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:20 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:20 smithi023 ceph-mon[36406]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:20 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3098651734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:20 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:20 smithi165 ceph-mon[36634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:20.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:20 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3098651734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:21.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:20 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T18:25:21.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:20 smithi002 ceph-mon[31328]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T18:25:21.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:20 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3098651734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:21.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:21 smithi165 ceph-mon[36634]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:21.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:21 smithi023 ceph-mon[36406]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:21 smithi002 ceph-mon[31328]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:23.823 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:23 smithi002 ceph-mon[31328]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-11T18:25:23.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:23 smithi165 ceph-mon[36634]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-11T18:25:23.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:23 smithi023 ceph-mon[36406]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-11T18:25:25.776 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:25 smithi002 ceph-mon[31328]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T18:25:25.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:25 smithi165 ceph-mon[36634]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T18:25:25.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:25 smithi023 ceph-mon[36406]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-11T18:25:27.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:27 smithi023 ceph-mon[36406]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T18:25:27.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:27 smithi165 ceph-mon[36634]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T18:25:28.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:27 smithi002 ceph-mon[31328]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-11T18:25:29.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:29 smithi002 ceph-mon[31328]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T18:25:29.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T18:25:29.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:29.859 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:29 smithi002 ceph-mon[31328]: Deploying daemon osd.3 on smithi002 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:29 smithi165 ceph-mon[36634]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:29 smithi165 ceph-mon[36634]: Deploying daemon osd.3 on smithi002 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:29 smithi023 ceph-mon[36406]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:29 smithi023 ceph-mon[36406]: Deploying daemon osd.3 on smithi002 2024-03-11T18:25:31.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:31 smithi002 ceph-mon[31328]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 889 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-11T18:25:31.904 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:31 smithi023 ceph-mon[36406]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 889 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-11T18:25:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:31 smithi165 ceph-mon[36634]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 889 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-11T18:25:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:33 smithi002 ceph-mon[31328]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T18:25:33.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:33 smithi023 ceph-mon[36406]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T18:25:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:33 smithi165 ceph-mon[36634]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-11T18:25:35.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:35 smithi165 ceph-mon[36634]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:35.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:35 smithi023 ceph-mon[36406]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:36.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:35 smithi002 ceph-mon[31328]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:37 smithi023 ceph-mon[36406]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:37 smithi165 ceph-mon[36634]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:38.001 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:37 smithi002 ceph-mon[31328]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:38.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:38.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:38.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:39.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:39 smithi002 ceph-mon[31328]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:39.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:39 smithi023 ceph-mon[36406]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:39.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:39 smithi165 ceph-mon[36634]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:41 smithi002 ceph-mon[31328]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T18:25:42.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:41 smithi165 ceph-mon[36634]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:41 smithi023 ceph-mon[36406]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T18:25:42.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.177+0000 7f4c37400700 1 -- 172.21.15.2:0/1278525003 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4c4002d9b0 con 0x7f4c240a1a30 2024-03-11T18:25:42.179 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 msgr2=0x7f4c240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 0x7f4c240a3ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c005fd0 tx=0x7f4c2c009310 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 msgr2=0x7f4c40199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e9a0 tx=0x7f4c3000ecb0 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 shutdown_connections 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c240a1a30 0x7f4c240a3ef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c40109bd0 0x7f4c401a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c40074670 0x7f4c40199f10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 --2- 172.21.15.2:0/1278525003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c40075c40 0x7f4c4019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:42.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.180+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 >> 172.21.15.2:0/1278525003 conn(0x7f4c400fd290 msgr2=0x7f4c40105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:25:42.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.181+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 shutdown_connections 2024-03-11T18:25:42.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:25:42.181+0000 7f4c45800700 1 -- 172.21.15.2:0/1278525003 wait complete. 2024-03-11T18:25:42.949 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T18:25:42.957 INFO:tasks.cephadm:Deploying osd.4 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-11T18:25:42.957 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T18:25:43.216 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:43.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T18:25:43.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T18:25:43.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:25:43.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:43.339 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.339 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.339 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:25:43.934 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:25:43.607+0000 7efc95a00700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T18:25:43.934 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:25:43.622+0000 7efc8b800700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:43 smithi002 ceph-mon[31328]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' 2024-03-11T18:25:44.316 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:44.316 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:25:44.316 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T18:25:44.316 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.317 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.317 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T18:25:44.317 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.317 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:43 smithi023 ceph-mon[36406]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:43 smithi165 ceph-mon[36634]: from='osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705]' entity='osd.3' 2024-03-11T18:25:45.238 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0105201 s, 997 MB/s 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 2024-03-11T18:25:45.239 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T18:25:45.239 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:25:45.239 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:25:45.240 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:45.240 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] boot 2024-03-11T18:25:45.240 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T18:25:45.240 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:44 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] boot 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T18:25:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:44 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: osd.3 [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] boot 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T18:25:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:44 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:25:46.185 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-11T18:25:46.215 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:45 smithi023 ceph-mon[36406]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:46.215 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:45 smithi023 ceph-mon[36406]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T18:25:46.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:45 smithi002 ceph-mon[31328]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:46.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:45 smithi002 ceph-mon[31328]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T18:25:46.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:45 smithi165 ceph-mon[36634]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T18:25:46.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:45 smithi165 ceph-mon[36634]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T18:25:46.434 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:25:47.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:47 smithi165 ceph-mon[36634]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:47.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:47 smithi165 ceph-mon[36634]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T18:25:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:47 smithi023 ceph-mon[36406]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:47 smithi023 ceph-mon[36406]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T18:25:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:47 smithi002 ceph-mon[31328]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:47 smithi002 ceph-mon[31328]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T18:25:48.241 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:48 smithi023 ceph-mon[36406]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T18:25:48.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 -- 172.21.15.23:0/2300195793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac103840 msgr2=0x7facac10a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:48.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 --2- 172.21.15.23:0/2300195793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac103840 0x7facac10a0f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7faca4009fc0 tx=0x7faca400b330 comp rx=0 tx=0).stop 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 -- 172.21.15.23:0/2300195793 shutdown_connections 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 --2- 172.21.15.23:0/2300195793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac101ca0 0x7facac1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 --2- 172.21.15.23:0/2300195793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac103840 0x7facac10a0f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 --2- 172.21.15.23:0/2300195793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facac102ea0 0x7facac103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 -- 172.21.15.23:0/2300195793 >> 172.21.15.23:0/2300195793 conn(0x7facac0fd280 msgr2=0x7facac0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 -- 172.21.15.23:0/2300195793 shutdown_connections 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.242+0000 7facb1695700 1 -- 172.21.15.23:0/2300195793 wait complete. 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 Processor -- start 2024-03-11T18:25:48.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 -- start start 2024-03-11T18:25:48.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facac101ca0 0x7facac197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:48.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:48.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac103840 0x7facac19f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:48.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac19f8e0 con 0x7facac101ca0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac19fa20 con 0x7facac102ea0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facb1695700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facac19fbe0 con 0x7facac103840 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facaa7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facaa7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56200/0 (socket says 172.21.15.23:56200) 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.243+0000 7facaa7fc700 1 -- 172.21.15.23:0/1680126350 learned_addr learned my addr 172.21.15.23:0/1680126350 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facab7fe700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac103840 0x7facac19f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 -- 172.21.15.23:0/1680126350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac103840 msgr2=0x7facac19f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac103840 0x7facac19f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 -- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facac101ca0 msgr2=0x7facac197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facac101ca0 0x7facac197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 -- 172.21.15.23:0/1680126350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c0095f0 con 0x7facac102ea0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facaa7fc700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faca000b530 tx=0x7faca000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:25:48.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0011830 con 0x7facac102ea0 2024-03-11T18:25:48.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca4009c40 con 0x7facac102ea0 2024-03-11T18:25:48.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faca0011e70 con 0x7facac102ea0 2024-03-11T18:25:48.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca000f2e0 con 0x7facac102ea0 2024-03-11T18:25:48.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.244+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facac1a02a0 con 0x7facac102ea0 2024-03-11T18:25:48.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.245+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac109060 con 0x7facac102ea0 2024-03-11T18:25:48.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.246+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7faca00105f0 con 0x7facac102ea0 2024-03-11T18:25:48.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.247+0000 7fac93fff700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 0x7fac940a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:25:48.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.247+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3669+0+0 (secure 0 0 0) 0x7faca00cc170 con 0x7facac102ea0 2024-03-11T18:25:48.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.248+0000 7facaaffd700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 0x7fac940a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:25:48.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.249+0000 7facaaffd700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 0x7fac940a3f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0095c0 tx=0x7fac9c009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:25:48.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.251+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faca00965a0 con 0x7facac102ea0 2024-03-11T18:25:48.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:48 smithi165 ceph-mon[36634]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T18:25:48.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:25:48.465+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7facac02d870 con 0x7fac940a1ac0 2024-03-11T18:25:48.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:48 smithi002 ceph-mon[31328]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: osdmap e31: 4 total, 4 up, 4 in 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: osdmap e31: 4 total, 4 up, 4 in 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: osdmap e31: 4 total, 4 up, 4 in 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:25:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi002 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 8547M 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi002 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 8547M 2024-03-11T18:25:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:50.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:50.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi002 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:25:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 8547M 2024-03-11T18:25:50.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:50.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:50.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:25:50.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:25:51.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:51 smithi002 ceph-mon[31328]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-11T18:25:51.630 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:51 smithi023 ceph-mon[36406]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-11T18:25:51.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:51 smithi165 ceph-mon[36634]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-11T18:25:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/2056174247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]': finished 2024-03-11T18:25:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T18:25:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:25:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:52 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/2095897303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:52.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/2056174247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]': finished 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:52 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/2095897303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/2056174247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]: dispatch 2024-03-11T18:25:52.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fa522c2-c663-4c43-b022-ce54054575ab"}]': finished 2024-03-11T18:25:52.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T18:25:52.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:25:52.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:52 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/2095897303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:25:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:53 smithi165 ceph-mon[36634]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:25:53.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:53 smithi023 ceph-mon[36406]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:25:53.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:53 smithi002 ceph-mon[31328]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:25:55.565 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:55 smithi023 ceph-mon[36406]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:55 smithi165 ceph-mon[36634]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:55.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:55 smithi002 ceph-mon[31328]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:57.550 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:57 smithi023 ceph-mon[36406]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:57.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:57 smithi165 ceph-mon[36634]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:57 smithi002 ceph-mon[31328]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:59.547 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:59 smithi023 ceph-mon[36406]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:59.548 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:59 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T18:25:59.548 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:59 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:59.548 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:25:59 smithi023 ceph-mon[36406]: Deploying daemon osd.4 on smithi023 2024-03-11T18:25:59.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:59 smithi002 ceph-mon[31328]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:59.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:59 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T18:25:59.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:59 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:59.555 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:25:59 smithi002 ceph-mon[31328]: Deploying daemon osd.4 on smithi023 2024-03-11T18:25:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:59 smithi165 ceph-mon[36634]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:25:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:59 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T18:25:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:59 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:25:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:25:59 smithi165 ceph-mon[36634]: Deploying daemon osd.4 on smithi023 2024-03-11T18:26:01.471 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:01 smithi023 ceph-mon[36406]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:01 smithi002 ceph-mon[31328]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:01.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:01 smithi165 ceph-mon[36634]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:03.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:03 smithi165 ceph-mon[36634]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:03 smithi023 ceph-mon[36406]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:03 smithi002 ceph-mon[31328]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:05 smithi002 ceph-mon[31328]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:06.063 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:05 smithi023 ceph-mon[36406]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:06.064 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:06.064 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:06.064 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:05 smithi165 ceph-mon[36634]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:08.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:08.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.501+0000 7fac93fff700 1 -- 172.21.15.23:0/1680126350 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7facac02d870 con 0x7fac940a1ac0 2024-03-11T18:26:08.502 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 4 on host 'smithi023' 2024-03-11T18:26:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 msgr2=0x7fac940a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 0x7fac940a3f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0095c0 tx=0x7fac9c009510 comp rx=0 tx=0).stop 2024-03-11T18:26:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 msgr2=0x7facac1982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faca000b530 tx=0x7faca000b8f0 comp rx=0 tx=0).stop 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 shutdown_connections 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fac940a1ac0 0x7fac940a3f80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facac103840 0x7facac19f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facac101ca0 0x7facac197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 --2- 172.21.15.23:0/1680126350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facac102ea0 0x7facac1982d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 >> 172.21.15.23:0/1680126350 conn(0x7facac0fd280 msgr2=0x7facac1061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 shutdown_connections 2024-03-11T18:26:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:26:08.503+0000 7facb1695700 1 -- 172.21.15.23:0/1680126350 wait complete. 2024-03-11T18:26:09.138 DEBUG:teuthology.orchestra.run.smithi023:osd.4> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T18:26:09.180 INFO:tasks.cephadm:Deploying osd.5 on smithi165 with /dev/vg_nvme/lv_3... 2024-03-11T18:26:09.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T18:26:09.427 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:26:09.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:09.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:09.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:09.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:09 smithi002 ceph-mon[31328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:09.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:09.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:09.766 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:09.767 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:09 smithi165 ceph-mon[36634]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:09 smithi023 ceph-mon[36406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T18:26:10.650 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:10 smithi165 ceph-mon[36634]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T18:26:10.650 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:10 smithi165 ceph-mon[36634]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:10.650 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:10 smithi165 ceph-mon[36634]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T18:26:10.650 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:10 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:10.650 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:10 smithi165 ceph-mon[36634]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-mon[36406]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T18:26:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-mon[36406]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-mon[36406]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T18:26:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-mon[36406]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:10.655 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:26:10.506+0000 7f42df06a700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T18:26:10.655 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:26:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:26:10.515+0000 7f42dae5d700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:26:11.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:10 smithi002 ceph-mon[31328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T18:26:11.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:10 smithi002 ceph-mon[31328]: from='osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:11.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:10 smithi002 ceph-mon[31328]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T18:26:11.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:10 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:10 smithi002 ceph-mon[31328]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:26:11.201 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T18:26:11.201 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T18:26:11.201 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-11T18:26:11.202 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-11T18:26:11.202 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0156386 s, 671 MB/s 2024-03-11T18:26:11.202 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-11T18:26:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:11 smithi002 ceph-mon[31328]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:11 smithi002 ceph-mon[31328]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:26:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:11 smithi002 ceph-mon[31328]: osdmap e34: 5 total, 4 up, 5 in 2024-03-11T18:26:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.792 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:11 smithi165 ceph-mon[36634]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:11.793 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:11 smithi165 ceph-mon[36634]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:26:11.793 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:11 smithi165 ceph-mon[36634]: osdmap e34: 5 total, 4 up, 5 in 2024-03-11T18:26:11.793 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.793 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-03-11T18:26:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:11 smithi023 ceph-mon[36406]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:11 smithi023 ceph-mon[36406]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:26:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:11 smithi023 ceph-mon[36406]: osdmap e34: 5 total, 4 up, 5 in 2024-03-11T18:26:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:12.066 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] boot 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:12.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] boot 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:12.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:13.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:26:13.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: osd.4 [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] boot 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:13.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:13.698 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 -- 172.21.15.165:0/4037261219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 msgr2=0x7f7980102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 --2- 172.21.15.165:0/4037261219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f7980102270 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7970009990 tx=0x7f7970009ca0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 -- 172.21.15.165:0/4037261219 shutdown_connections 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 --2- 172.21.15.165:0/4037261219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79801027b0 0x7f7980109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 --2- 172.21.15.165:0/4037261219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7980100c10 0x7f7980101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 --2- 172.21.15.165:0/4037261219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f7980102270 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.696+0000 7f79878c1700 1 -- 172.21.15.165:0/4037261219 >> 172.21.15.165:0/4037261219 conn(0x7f79800fc1d0 msgr2=0x7f79800fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- 172.21.15.165:0/4037261219 shutdown_connections 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- 172.21.15.165:0/4037261219 wait complete. 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 Processor -- start 2024-03-11T18:26:13.699 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- start start 2024-03-11T18:26:13.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7980100c10 0x7f7980104b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:13.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:13.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79801027b0 0x7f7980103730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:13.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980103ed0 con 0x7f79801027b0 2024-03-11T18:26:13.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980104040 con 0x7f7980100c10 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f79878c1700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79801a1e80 con 0x7f7980101e10 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f7984e5c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f7984e5c700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:35504/0 (socket says 172.21.15.165:35504) 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.697+0000 7f7984e5c700 1 -- 172.21.15.165:0/4079819897 learned_addr learned my addr 172.21.15.165:0/4079819897 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f798565d700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7980100c10 0x7f7980104b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 -- 172.21.15.165:0/4079819897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7980100c10 msgr2=0x7f7980104b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7980100c10 0x7f7980104b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 -- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79801027b0 msgr2=0x7f7980103730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79801027b0 0x7f7980103730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 -- 172.21.15.165:0/4079819897 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79700095f0 con 0x7f7980101e10 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f7984e5c700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7970005910 tx=0x7f797000f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797001d070 con 0x7f7980101e10 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f797000fc90 con 0x7f7980101e10 2024-03-11T18:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79801a20b0 con 0x7f7980101e10 2024-03-11T18:26:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970017770 con 0x7f7980101e10 2024-03-11T18:26:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.698+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79801a2540 con 0x7f7980101e10 2024-03-11T18:26:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.699+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7980068930 con 0x7f7980101e10 2024-03-11T18:26:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.700+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f7970022a50 con 0x7f7980101e10 2024-03-11T18:26:13.703 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.701+0000 7f79767fc700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 0x7f796c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:13.704 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.701+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f79700cd8e0 con 0x7f7980101e10 2024-03-11T18:26:13.704 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.702+0000 7f798565d700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 0x7f796c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:13.704 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.703+0000 7f798565d700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 0x7f796c0a3f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f797c006fd0 tx=0x7f797c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:26:13.706 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.705+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f797005dd20 con 0x7f7980101e10 2024-03-11T18:26:13.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:13 smithi165 ceph-mon[36634]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:13.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:13 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi023 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:13 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 8540M 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:13 smithi165 ceph-mon[36634]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:13 smithi023 ceph-mon[36406]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:13 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi023 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:13 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 8540M 2024-03-11T18:26:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:13 smithi023 ceph-mon[36406]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T18:26:13.906 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:13.904+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f79800fe3d0 con 0x7f796c0a1ac0 2024-03-11T18:26:14.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:13 smithi002 ceph-mon[31328]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-11T18:26:14.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:13 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi023 2024-03-11T18:26:14.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:13 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 8540M 2024-03-11T18:26:14.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:13 smithi002 ceph-mon[31328]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T18:26:14.805 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:14 smithi165 ceph-mon[36634]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:14.805 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:14.805 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:14.805 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:14 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:14.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:14 smithi023 ceph-mon[36406]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:14.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:14.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:14.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:14 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:14 smithi002 ceph-mon[31328]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:15.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:14 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:15.787 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:15 smithi165 ceph-mon[36634]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:15.904 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:15 smithi023 ceph-mon[36406]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:16.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:15 smithi002 ceph-mon[31328]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/2359398183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]': finished 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/2733325064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:17.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/2359398183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]': finished 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/2733325064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/2359398183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]: dispatch 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0f4c30d-2ccb-4ed3-8562-a935e88af3b9"}]': finished 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T18:26:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:18.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/2733325064' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:19.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:19 smithi165 ceph-mon[36634]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:19.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:19 smithi023 ceph-mon[36406]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:19 smithi002 ceph-mon[31328]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:21.827 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:21 smithi165 ceph-mon[36634]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:21.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:21 smithi023 ceph-mon[36406]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:21 smithi002 ceph-mon[31328]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:23.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:23 smithi165 ceph-mon[36634]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:23.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:23 smithi023 ceph-mon[36406]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:24.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:23 smithi002 ceph-mon[31328]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:24.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T18:26:24.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:24.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:24 smithi165 ceph-mon[36634]: Deploying daemon osd.5 on smithi165 2024-03-11T18:26:24.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T18:26:24.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:24.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:24 smithi023 ceph-mon[36406]: Deploying daemon osd.5 on smithi165 2024-03-11T18:26:25.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T18:26:25.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:25.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:24 smithi002 ceph-mon[31328]: Deploying daemon osd.5 on smithi165 2024-03-11T18:26:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:25 smithi165 ceph-mon[36634]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:25.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:25 smithi023 ceph-mon[36406]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:25 smithi002 ceph-mon[31328]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:27.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:27 smithi023 ceph-mon[36406]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:27.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:27 smithi165 ceph-mon[36634]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:28.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:27 smithi002 ceph-mon[31328]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:29.877 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:29 smithi165 ceph-mon[36634]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:29.877 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:29.877 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:29.877 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:29 smithi023 ceph-mon[36406]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:29.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:29 smithi002 ceph-mon[31328]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:31.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:31 smithi165 ceph-mon[36634]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:31.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:31 smithi023 ceph-mon[36406]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:32.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:31 smithi002 ceph-mon[31328]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:32.643 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.641+0000 7f79767fc700 1 -- 172.21.15.165:0/4079819897 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f79800fe3d0 con 0x7f796c0a1ac0 2024-03-11T18:26:32.643 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 5 on host 'smithi165' 2024-03-11T18:26:32.645 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 msgr2=0x7f796c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:32.645 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 0x7f796c0a3f80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f797c006fd0 tx=0x7f797c008040 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 msgr2=0x7f79801031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7970005910 tx=0x7f797000f7b0 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 shutdown_connections 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79801027b0 0x7f7980103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7980100c10 0x7f7980104b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f796c0a1ac0 0x7f796c0a3f80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 --2- 172.21.15.165:0/4079819897 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7980101e10 0x7f79801031f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:32.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 >> 172.21.15.165:0/4079819897 conn(0x7f79800fc1d0 msgr2=0x7f79800fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:26:32.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 shutdown_connections 2024-03-11T18:26:32.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:26:32.644+0000 7f79878c1700 1 -- 172.21.15.165:0/4079819897 wait complete. 2024-03-11T18:26:33.302 DEBUG:teuthology.orchestra.run.smithi165:osd.5> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T18:26:33.304 INFO:tasks.cephadm:Deploying osd.6 on smithi002 with /dev/vg_nvme/lv_2... 2024-03-11T18:26:33.305 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:33 smithi002 ceph-mon[31328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:33.784 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:33 smithi165 ceph-mon[36634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:33 smithi023 ceph-mon[36406]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T18:26:34.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-mon[36634]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T18:26:34.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-mon[36634]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T18:26:34.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:34.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-mon[36634]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:34.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-mon[36634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:34.906 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:26:34.763+0000 7f6f3ae4d700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T18:26:34.906 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:26:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:26:34.769+0000 7f6f36c40700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:26:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:34 smithi023 ceph-mon[36406]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T18:26:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:34 smithi023 ceph-mon[36406]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T18:26:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:34 smithi023 ceph-mon[36406]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:34 smithi023 ceph-mon[36406]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:35.161 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:34 smithi002 ceph-mon[31328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T18:26:35.161 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:34 smithi002 ceph-mon[31328]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T18:26:35.161 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:35.161 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:34 smithi002 ceph-mon[31328]: from='osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:35.161 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:34 smithi002 ceph-mon[31328]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:26:36.077 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:35 smithi165 ceph-mon[36634]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:36.077 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:35 smithi165 ceph-mon[36634]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:26:36.077 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:35 smithi165 ceph-mon[36634]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T18:26:36.077 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:36.077 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:35 smithi165 ceph-mon[36634]: osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] boot 2024-03-11T18:26:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:35 smithi023 ceph-mon[36406]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:35 smithi023 ceph-mon[36406]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:26:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:35 smithi023 ceph-mon[36406]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T18:26:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:35 smithi023 ceph-mon[36406]: osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] boot 2024-03-11T18:26:36.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:35 smithi002 ceph-mon[31328]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-11T18:26:36.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:35 smithi002 ceph-mon[31328]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:26:36.209 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:35 smithi002 ceph-mon[31328]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T18:26:36.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:36.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:35 smithi002 ceph-mon[31328]: osd.5 [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] boot 2024-03-11T18:26:36.698 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T18:26:36.698 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T18:26:36.698 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-03-11T18:26:36.698 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-03-11T18:26:36.698 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0323482 s, 324 MB/s 2024-03-11T18:26:36.699 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-03-11T18:26:36.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:26:36.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:26:36.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T18:26:36.870 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:36.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:36 smithi002 ceph-mon[31328]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:36 smithi165 ceph-mon[36634]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:37.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:26:37.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:37.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:26:37.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:26:37.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:36 smithi023 ceph-mon[36406]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T18:26:37.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2024-03-11T18:26:37.968 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:26:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:37 smithi002 ceph-mon[31328]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:37 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi165 2024-03-11T18:26:38.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:37 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 8540M 2024-03-11T18:26:38.116 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:37 smithi002 ceph-mon[31328]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T18:26:38.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:37 smithi165 ceph-mon[36634]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:38.154 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:37 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi165 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:37 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 8540M 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:37 smithi165 ceph-mon[36634]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:37 smithi023 ceph-mon[36406]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:37 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi165 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:37 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 8540M 2024-03-11T18:26:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:37 smithi023 ceph-mon[36406]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T18:26:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:39 smithi002 ceph-mon[31328]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T18:26:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:39 smithi002 ceph-mon[31328]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:39.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:39 smithi023 ceph-mon[36406]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T18:26:39.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:39 smithi023 ceph-mon[36406]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:39 smithi165 ceph-mon[36634]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T18:26:39.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:39 smithi165 ceph-mon[36634]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:40.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:40 smithi002 ceph-mon[31328]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.231+0000 7f3279a00700 1 -- 172.21.15.2:0/2534415979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 msgr2=0x7f3274103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.231+0000 7f3279a00700 1 --2- 172.21.15.2:0/2534415979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274103220 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f325c0099c0 tx=0x7f325c009cd0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 -- 172.21.15.2:0/2534415979 shutdown_connections 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 --2- 172.21.15.2:0/2534415979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274101bc0 0x7f3274101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 --2- 172.21.15.2:0/2534415979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274103760 0x7f327410a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 --2- 172.21.15.2:0/2534415979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274103220 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 -- 172.21.15.2:0/2534415979 >> 172.21.15.2:0/2534415979 conn(0x7f32740fd180 msgr2=0x7f32740ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:26:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 -- 172.21.15.2:0/2534415979 shutdown_connections 2024-03-11T18:26:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.232+0000 7f3279a00700 1 -- 172.21.15.2:0/2534415979 wait complete. 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.233+0000 7f3279a00700 1 Processor -- start 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 -- start start 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274103760 0x7f32740745a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3274074d20 con 0x7f3274102dc0 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3274074e90 con 0x7f3274101bc0 2024-03-11T18:26:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3279a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32741a1bb0 con 0x7f3274103760 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f326b400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.234+0000 7f3273e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f326b400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50576/0 (socket says 172.21.15.2:50576) 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f326b400700 1 -- 172.21.15.2:0/760739408 learned_addr learned my addr 172.21.15.2:0/760739408 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55256/0 (socket says 172.21.15.2:55256) 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3279000700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274103760 0x7f32740745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 -- 172.21.15.2:0/760739408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274103760 msgr2=0x7f32740745a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274103760 0x7f32740745a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 -- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 msgr2=0x7f3274074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 -- 172.21.15.2:0/760739408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0095f0 con 0x7f3274101bc0 2024-03-11T18:26:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f326b400700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.235+0000 7f3273e00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f326400e9c0 tx=0x7f326400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.236+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264004220 con 0x7f3274101bc0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.236+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f326400cca0 con 0x7f3274101bc0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.236+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264003c50 con 0x7f3274101bc0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.236+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32741a1e90 con 0x7f3274101bc0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.236+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32741a22a0 con 0x7f3274101bc0 2024-03-11T18:26:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.237+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3274108f20 con 0x7f3274101bc0 2024-03-11T18:26:40.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.239+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3264004380 con 0x7f3274101bc0 2024-03-11T18:26:40.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.240+0000 7f3271600700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 0x7f32580a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:26:40.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.240+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f3264014070 con 0x7f3274101bc0 2024-03-11T18:26:40.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.241+0000 7f326b400700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 0x7f32580a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:26:40.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.241+0000 7f326b400700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 0x7f32580a3f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f325c0036c0 tx=0x7f325c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:26:40.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.245+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3264096800 con 0x7f3274101bc0 2024-03-11T18:26:40.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:40 smithi165 ceph-mon[36634]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T18:26:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:40 smithi023 ceph-mon[36406]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T18:26:40.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:26:40.482+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f327402d950 con 0x7f32580a1ac0 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:41 smithi165 ceph-mon[36634]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:41 smithi165 ceph-mon[36634]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:41 smithi023 ceph-mon[36406]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:41 smithi023 ceph-mon[36406]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:41.460 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:41 smithi002 ceph-mon[31328]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:41.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:41 smithi002 ceph-mon[31328]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:26:41.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:26:41.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:26:41.461 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:43.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:43 smithi002 ceph-mon[31328]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T18:26:43.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:43 smithi165 ceph-mon[36634]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T18:26:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:43 smithi023 ceph-mon[36406]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:45 smithi165 ceph-mon[36634]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:45 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]: dispatch 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:45 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]': finished 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:45 smithi165 ceph-mon[36634]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:26:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:45 smithi023 ceph-mon[36406]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T18:26:45.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:45 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]: dispatch 2024-03-11T18:26:45.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:45 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]': finished 2024-03-11T18:26:45.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:45 smithi023 ceph-mon[36406]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T18:26:45.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:26:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:45 smithi002 ceph-mon[31328]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-11T18:26:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]: dispatch 2024-03-11T18:26:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2641927109' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1f9bcec-338a-44b6-bddb-920dac2a46e0"}]': finished 2024-03-11T18:26:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:45 smithi002 ceph-mon[31328]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T18:26:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:26:46.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/801151145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/801151145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/801151145' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:26:47.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:47 smithi165 ceph-mon[36634]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:26:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:47 smithi023 ceph-mon[36406]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:26:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:47 smithi002 ceph-mon[31328]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:26:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:49 smithi002 ceph-mon[31328]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:26:49.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:49 smithi165 ceph-mon[36634]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:26:49.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:49 smithi023 ceph-mon[36406]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:26:51.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:51 smithi002 ceph-mon[31328]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T18:26:51.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:51 smithi165 ceph-mon[36634]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T18:26:51.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:51 smithi023 ceph-mon[36406]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T18:26:53.453 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:53 smithi002 ceph-mon[31328]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:53.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:53 smithi023 ceph-mon[36406]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:53 smithi165 ceph-mon[36634]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:55.429 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:55 smithi002 ceph-mon[31328]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:55.430 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:55 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T18:26:55.430 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:55 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:55.430 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:55 smithi002 ceph-mon[31328]: Deploying daemon osd.6 on smithi002 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:55 smithi165 ceph-mon[36634]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:55 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:55 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:55 smithi165 ceph-mon[36634]: Deploying daemon osd.6 on smithi002 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:55 smithi023 ceph-mon[36406]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:55.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:55 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T18:26:55.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:55 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:26:55.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:55 smithi023 ceph-mon[36406]: Deploying daemon osd.6 on smithi002 2024-03-11T18:26:57.453 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:57 smithi002 ceph-mon[31328]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:57 smithi165 ceph-mon[36634]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:57 smithi023 ceph-mon[36406]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:59.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:26:59 smithi002 ceph-mon[31328]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:59.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:26:59 smithi023 ceph-mon[36406]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:26:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:26:59 smithi165 ceph-mon[36634]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:01.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:01 smithi002 ceph-mon[31328]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:01.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:01 smithi165 ceph-mon[36634]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:01.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:01 smithi023 ceph-mon[36406]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:03.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:03 smithi002 ceph-mon[31328]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:03.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:03 smithi165 ceph-mon[36634]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:03 smithi023 ceph-mon[36406]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:04.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:04.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:04.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:05.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:05 smithi002 ceph-mon[31328]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:05 smithi165 ceph-mon[36634]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:05.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:05 smithi023 ceph-mon[36406]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:07.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:07 smithi165 ceph-mon[36634]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:07.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.707 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:07 smithi165 ceph-mon[36634]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T18:27:07.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:07 smithi002 ceph-mon[31328]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:07.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:07 smithi002 ceph-mon[31328]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T18:27:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:07 smithi023 ceph-mon[36406]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:07 smithi023 ceph-mon[36406]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T18:27:08.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.237+0000 7f3271600700 1 -- 172.21.15.2:0/760739408 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f327402d950 con 0x7f32580a1ac0 2024-03-11T18:27:08.239 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 6 on host 'smithi002' 2024-03-11T18:27:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 msgr2=0x7f32580a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 0x7f32580a3f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f325c0036c0 tx=0x7f325c005fb0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 msgr2=0x7f3274075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f326400e9c0 tx=0x7f326400ecd0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 shutdown_connections 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32580a1ac0 0x7f32580a3f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3274103760 0x7f32740745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274101bc0 0x7f3274075a10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 --2- 172.21.15.2:0/760739408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3274102dc0 0x7f3274074060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.240+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 >> 172.21.15.2:0/760739408 conn(0x7f32740fd180 msgr2=0x7f3274106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:27:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.241+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 shutdown_connections 2024-03-11T18:27:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:27:08.241+0000 7f3279a00700 1 -- 172.21.15.2:0/760739408 wait complete. 2024-03-11T18:27:09.106 DEBUG:teuthology.orchestra.run.smithi002:osd.6> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T18:27:09.110 INFO:tasks.cephadm:Deploying osd.7 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-11T18:27:09.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T18:27:09.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:09.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T18:27:09.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T18:27:09.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:27:09.342 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:09.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:09.343 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:09.372 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:09.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:10.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:10 smithi002 ceph-mon[31328]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:27:10.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:10 smithi002 ceph-mon[31328]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T18:27:10.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:10 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:10.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:10 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:10.614 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:10 smithi023 ceph-mon[36406]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:27:10.614 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:10 smithi023 ceph-mon[36406]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T18:27:10.614 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:10 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:10.614 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:10 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:10.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:10 smithi165 ceph-mon[36634]: from='osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:27:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:10 smithi165 ceph-mon[36634]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T18:27:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:10 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:10 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] boot 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T18:27:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] boot 2024-03-11T18:27:11.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T18:27:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.535 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T18:27:11.535 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T18:27:11.536 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T18:27:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T18:27:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152163 s, 689 MB/s 2024-03-11T18:27:11.536 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: osd.6 [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] boot 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T18:27:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:27:12.269 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-11T18:27:12.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:12 smithi002 ceph-mon[31328]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T18:27:12.520 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:27:12.643 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:12 smithi023 ceph-mon[36406]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T18:27:12.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:12 smithi165 ceph-mon[36634]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T18:27:13.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:13 smithi002 ceph-mon[31328]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:13.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:13 smithi023 ceph-mon[36406]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:13.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:13 smithi165 ceph-mon[36634]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:14.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 -- 172.21.15.23:0/2915668227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4107990 msgr2=0x7f61d4109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:14.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 --2- 172.21.15.23:0/2915668227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4107990 0x7f61d4109e00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00b0a0 tx=0x7f61cc00b3b0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 -- 172.21.15.23:0/2915668227 shutdown_connections 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 --2- 172.21.15.23:0/2915668227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4107990 0x7f61d4109e00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 --2- 172.21.15.23:0/2915668227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4102730 0x7f61d4104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 --2- 172.21.15.23:0/2915668227 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4105060 0x7f61d4107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 -- 172.21.15.23:0/2915668227 >> 172.21.15.23:0/2915668227 conn(0x7f61d40fc160 msgr2=0x7f61d40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 -- 172.21.15.23:0/2915668227 shutdown_connections 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.347+0000 7f61d9fab700 1 -- 172.21.15.23:0/2915668227 wait complete. 2024-03-11T18:27:14.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 Processor -- start 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 -- start start 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4105060 0x7f61d41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4107990 0x7f61d419f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d419f8f0 con 0x7f61d4105060 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d419fa30 con 0x7f61d4102730 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d9fab700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d419fbc0 con 0x7f61d4107990 2024-03-11T18:27:14.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:14.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42466/0 (socket says 172.21.15.23:42466) 2024-03-11T18:27:14.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.348+0000 7f61d37fe700 1 -- 172.21.15.23:0/4265914255 learned_addr learned my addr 172.21.15.23:0/4265914255 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-11T18:27:14.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d2ffd700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4105060 0x7f61d41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:14.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d3fff700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4107990 0x7f61d419f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 -- 172.21.15.23:0/4265914255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4107990 msgr2=0x7f61d419f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4107990 0x7f61d419f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 -- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4105060 msgr2=0x7f61d41982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4105060 0x7f61d41982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 -- 172.21.15.23:0/4265914255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc009d00 con 0x7f61d4102730 2024-03-11T18:27:14.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.349+0000 7f61d37fe700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61c400d950 tx=0x7f61c400dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.350+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c4009b40 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.350+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d419fea0 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.350+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61c4009ca0 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.350+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c400f6b0 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.350+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d41a02c0 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.351+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d4068930 con 0x7f61d4102730 2024-03-11T18:27:14.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.352+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f61c4010460 con 0x7f61d4102730 2024-03-11T18:27:14.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.352+0000 7f61d0ff9700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 0x7f61bc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:14.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.352+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f61c40cc5e0 con 0x7f61d4102730 2024-03-11T18:27:14.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.353+0000 7f61d2ffd700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 0x7f61bc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:14.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.353+0000 7f61d2ffd700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 0x7f61bc0a3f80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005950 tx=0x7f61c80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:27:14.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.357+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61c4096500 con 0x7f61d4102730 2024-03-11T18:27:14.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:14.572+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d402d990 con 0x7f61bc0a1ac0 2024-03-11T18:27:15.486 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:15.486 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi002 2024-03-11T18:27:15.486 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.486 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 5698M 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:15.487 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.488 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:15.488 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:15.488 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi002 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 5698M 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:15.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi002 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 5698M 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:15.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:15.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:15.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:17.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:17 smithi002 ceph-mon[31328]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:17.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:17 smithi165 ceph-mon[36634]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:17.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:17 smithi023 ceph-mon[36406]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/736964912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]': finished 2024-03-11T18:27:18.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T18:27:18.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:18.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/85630352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:18.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/736964912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]': finished 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:18 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/85630352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/736964912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]: dispatch 2024-03-11T18:27:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "787a342e-4f3a-47f0-9ff8-cf7e80400e7d"}]': finished 2024-03-11T18:27:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: osdmap e50: 8 total, 7 up, 8 in 2024-03-11T18:27:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:18 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/85630352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:19.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:19 smithi165 ceph-mon[36634]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:19.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:19 smithi002 ceph-mon[31328]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:19.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:19 smithi023 ceph-mon[36406]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:21.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:21 smithi002 ceph-mon[31328]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:21.809 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:21 smithi023 ceph-mon[36406]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:21.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:21 smithi165 ceph-mon[36634]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:23.733 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:23 smithi023 ceph-mon[36406]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:23.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:23 smithi002 ceph-mon[31328]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:23.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:23 smithi165 ceph-mon[36634]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:25.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:25 smithi002 ceph-mon[31328]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:25.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T18:27:25.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:25.790 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:25 smithi023 ceph-mon[36406]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:25.790 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T18:27:25.790 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:25 smithi165 ceph-mon[36634]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T18:27:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:26.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:26 smithi002 ceph-mon[31328]: Deploying daemon osd.7 on smithi023 2024-03-11T18:27:26.836 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:26 smithi023 ceph-mon[36406]: Deploying daemon osd.7 on smithi023 2024-03-11T18:27:26.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:26 smithi165 ceph-mon[36634]: Deploying daemon osd.7 on smithi023 2024-03-11T18:27:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:27 smithi002 ceph-mon[31328]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:27.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:27 smithi165 ceph-mon[36634]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:27.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:27 smithi023 ceph-mon[36406]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:29 smithi023 ceph-mon[36406]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:29.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:29 smithi002 ceph-mon[31328]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:29.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:29 smithi165 ceph-mon[36634]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:31.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:31 smithi002 ceph-mon[31328]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:31.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:31 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:31 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:31 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:31.826 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:31 smithi023 ceph-mon[36406]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:31.826 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:31 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.826 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:31 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.827 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:31 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:31 smithi165 ceph-mon[36634]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:31 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:31 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:31 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:33 smithi002 ceph-mon[31328]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:33.763 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:33 smithi023 ceph-mon[36406]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:33.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:33 smithi165 ceph-mon[36634]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:34.915 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 7 on host 'smithi023' 2024-03-11T18:27:34.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.913+0000 7f61d0ff9700 1 -- 172.21.15.23:0/4265914255 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61d402d990 con 0x7f61bc0a1ac0 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 msgr2=0x7f61bc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 0x7f61bc0a3f80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f61c8005950 tx=0x7f61c80058e0 comp rx=0 tx=0).stop 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 msgr2=0x7f61d4197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61c400d950 tx=0x7f61c400dd10 comp rx=0 tx=0).stop 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 shutdown_connections 2024-03-11T18:27:34.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61bc0a1ac0 0x7f61bc0a3f80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61d4107990 0x7f61d419f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61d4102730 0x7f61d4197d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 --2- 172.21.15.23:0/4265914255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61d4105060 0x7f61d41982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 >> 172.21.15.23:0/4265914255 conn(0x7f61d40fc160 msgr2=0x7f61d4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 shutdown_connections 2024-03-11T18:27:34.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:27:34.916+0000 7f61d9fab700 1 -- 172.21.15.23:0/4265914255 wait complete. 2024-03-11T18:27:35.693 DEBUG:teuthology.orchestra.run.smithi023:osd.7> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T18:27:35.696 INFO:tasks.cephadm:Deploying osd.8 on smithi165 with /dev/vg_nvme/lv_2... 2024-03-11T18:27:35.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T18:27:35.950 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.072 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T18:27:36.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:36.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:36 smithi165 ceph-mon[36634]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T18:27:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:36 smithi165 ceph-mon[36634]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T18:27:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:36 smithi165 ceph-mon[36634]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:36 smithi165 ceph-mon[36634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-mon[36406]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T18:27:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-mon[36406]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T18:27:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-mon[36406]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-mon[36406]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.156 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:27:36.909+0000 7fa4e6073700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T18:27:37.156 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:27:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:27:36.918+0000 7fa4e1e66700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:27:37.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:36 smithi002 ceph-mon[31328]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T18:27:37.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:36 smithi002 ceph-mon[31328]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T18:27:37.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:37.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:36 smithi002 ceph-mon[31328]: from='osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:36 smithi002 ceph-mon[31328]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:27:37.998 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T18:27:37.999 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T18:27:37.999 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-11T18:27:37.999 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-11T18:27:37.999 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117805 s, 890 MB/s 2024-03-11T18:27:37.999 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] boot 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T18:27:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:37 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] boot 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T18:27:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:37 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-11T18:27:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:27:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T18:27:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: osd.7 [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] boot 2024-03-11T18:27:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: osdmap e53: 8 total, 8 up, 8 in 2024-03-11T18:27:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:37 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:27:38.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-03-11T18:27:38.869 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:27:39.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:38 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:27:39.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:38 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:27:39.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:38 smithi023 ceph-mon[36406]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T18:27:39.211 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:38 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:27:39.212 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:38 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:27:39.212 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:38 smithi165 ceph-mon[36634]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T18:27:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:38 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:27:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:38 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:27:39.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:38 smithi002 ceph-mon[31328]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T18:27:39.965 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:39.966 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T18:27:39.966 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:39.966 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:39.966 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:39.966 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:39.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:39.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:39.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:39.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:39.967 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:39.968 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:39.968 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:39.968 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:39 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:40.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T18:27:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:40.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:40.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:39 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:27:40.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:40.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:40.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:40.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:40.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:40.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:39 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:27:40.568 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 -- 172.21.15.165:0/1349336309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c41050c0 msgr2=0x7f57c41074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:40.569 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 --2- 172.21.15.165:0/1349336309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c41050c0 0x7f57c41074b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009990 tx=0x7f57b8009ca0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.569 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 -- 172.21.15.165:0/1349336309 shutdown_connections 2024-03-11T18:27:40.569 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 --2- 172.21.15.165:0/1349336309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c41079f0 0x7f57c4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.569 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 --2- 172.21.15.165:0/1349336309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c4102790 0x7f57c4104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.569 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 --2- 172.21.15.165:0/1349336309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c41050c0 0x7f57c41074b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 -- 172.21.15.165:0/1349336309 >> 172.21.15.165:0/1349336309 conn(0x7f57c40fc140 msgr2=0x7f57c40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 -- 172.21.15.165:0/1349336309 shutdown_connections 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.562+0000 7f57c967b700 1 -- 172.21.15.165:0/1349336309 wait complete. 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 Processor -- start 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 -- start start 2024-03-11T18:27:40.570 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:40.571 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c41050c0 0x7f57c4100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:40.571 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c41079f0 0x7f57c4100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:40.571 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4101400 con 0x7f57c41050c0 2024-03-11T18:27:40.571 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4101570 con 0x7f57c41079f0 2024-03-11T18:27:40.571 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c967b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c41a1d50 con 0x7f57c4102790 2024-03-11T18:27:40.572 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.563+0000 7f57c2ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:40.572 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:56206/0 (socket says 172.21.15.165:56206) 2024-03-11T18:27:40.572 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 -- 172.21.15.165:0/2672727977 learned_addr learned my addr 172.21.15.165:0/2672727977 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:27:40.572 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c37fe700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c41079f0 0x7f57c4100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:40.572 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 -- 172.21.15.165:0/2672727977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c41079f0 msgr2=0x7f57c4100c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c41079f0 0x7f57c4100c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 -- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c41050c0 msgr2=0x7f57c4100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c41050c0 0x7f57c4100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 -- 172.21.15.165:0/2672727977 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b80095f0 con 0x7f57c4102790 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c2ffd700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f57b400c920 tx=0x7f57b400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:27:40.573 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4007b00 con 0x7f57c4102790 2024-03-11T18:27:40.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57b400ce70 con 0x7f57c4102790 2024-03-11T18:27:40.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c41a1fd0 con 0x7f57c4102790 2024-03-11T18:27:40.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4005610 con 0x7f57c4102790 2024-03-11T18:27:40.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.564+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41a2440 con 0x7f57c4102790 2024-03-11T18:27:40.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.565+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c4068930 con 0x7f57c4102790 2024-03-11T18:27:40.575 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.566+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f57b4004750 con 0x7f57c4102790 2024-03-11T18:27:40.575 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.567+0000 7f57abfff700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 0x7f57ac0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:27:40.575 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.567+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f57b40cc7f0 con 0x7f57c4102790 2024-03-11T18:27:40.575 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.567+0000 7f57c27fc700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 0x7f57ac0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:27:40.576 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.568+0000 7f57c27fc700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 0x7f57ac0a3f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098e0 tx=0x7f57b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:27:40.576 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.571+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57b4096560 con 0x7f57c4102790 2024-03-11T18:27:40.773 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:27:40.772+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f57c402d950 con 0x7f57ac0a1ac0 2024-03-11T18:27:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi023 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 5693M 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:41.402 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi023 2024-03-11T18:27:41.402 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 5693M 2024-03-11T18:27:41.403 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:41.403 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T18:27:41.403 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:41.403 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:41.403 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi023 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 5693M 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:27:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:41 smithi165 ceph-mon[36634]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:41 smithi165 ceph-mon[36634]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:41 smithi165 ceph-mon[36634]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T18:27:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:41 smithi002 ceph-mon[31328]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:41 smithi002 ceph-mon[31328]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:41 smithi002 ceph-mon[31328]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T18:27:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:41 smithi023 ceph-mon[36406]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:41 smithi023 ceph-mon[36406]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:27:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:41 smithi023 ceph-mon[36406]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T18:27:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/210739585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]': finished 2024-03-11T18:27:44.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T18:27:44.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/210739585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]': finished 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T18:27:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/210739585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]: dispatch 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e50d064-174a-482a-8643-898a23138c36"}]': finished 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: osdmap e58: 9 total, 8 up, 9 in 2024-03-11T18:27:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:27:45.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/4144730000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:45.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:44 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/4144730000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:44 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/4144730000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:27:46.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:45 smithi002 ceph-mon[31328]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:45 smithi023 ceph-mon[36406]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:46.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:45 smithi165 ceph-mon[36634]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:48.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:47 smithi002 ceph-mon[31328]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:48.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:47 smithi165 ceph-mon[36634]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:48.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:47 smithi023 ceph-mon[36406]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:50.207 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:49 smithi023 ceph-mon[36406]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:49 smithi002 ceph-mon[31328]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:50.253 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:49 smithi165 ceph-mon[36634]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:51.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T18:27:51.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:50 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:51.404 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T18:27:51.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:50 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:51.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-11T18:27:51.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:50 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:27:52.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:51 smithi165 ceph-mon[36634]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:52.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:51 smithi165 ceph-mon[36634]: Deploying daemon osd.8 on smithi165 2024-03-11T18:27:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:51 smithi002 ceph-mon[31328]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:51 smithi002 ceph-mon[31328]: Deploying daemon osd.8 on smithi165 2024-03-11T18:27:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:51 smithi023 ceph-mon[36406]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:51 smithi023 ceph-mon[36406]: Deploying daemon osd.8 on smithi165 2024-03-11T18:27:54.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:53 smithi002 ceph-mon[31328]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:54.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:53 smithi165 ceph-mon[36634]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:54.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:53 smithi023 ceph-mon[36406]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:56.097 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:55 smithi165 ceph-mon[36634]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:56.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:55 smithi002 ceph-mon[31328]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:56.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:55 smithi023 ceph-mon[36406]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:57.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:57 smithi165 ceph-mon[36634]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:57 smithi023 ceph-mon[36406]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:57 smithi002 ceph-mon[31328]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:27:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:27:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:27:59 smithi165 ceph-mon[36634]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:59.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:27:59 smithi023 ceph-mon[36406]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:27:59.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:27:59 smithi002 ceph-mon[31328]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:00.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.207+0000 7f57abfff700 1 -- 172.21.15.165:0/2672727977 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f57c402d950 con 0x7f57ac0a1ac0 2024-03-11T18:28:00.209 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 8 on host 'smithi165' 2024-03-11T18:28:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 msgr2=0x7f57ac0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 0x7f57ac0a3f80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57b80098e0 tx=0x7f57b8005fb0 comp rx=0 tx=0).stop 2024-03-11T18:28:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 msgr2=0x7f57c41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f57b400c920 tx=0x7f57b400cc30 comp rx=0 tx=0).stop 2024-03-11T18:28:00.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 shutdown_connections 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f57ac0a1ac0 0x7f57ac0a3f80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57c4102790 0x7f57c41020f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57c41079f0 0x7f57c4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 --2- 172.21.15.165:0/2672727977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c41050c0 0x7f57c4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 >> 172.21.15.165:0/2672727977 conn(0x7f57c40fc140 msgr2=0x7f57c41030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 shutdown_connections 2024-03-11T18:28:00.212 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:28:00.210+0000 7f57c967b700 1 -- 172.21.15.165:0/2672727977 wait complete. 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.519 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.520 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.520 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:00.520 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:00.520 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:00 smithi165 ceph-mon[36634]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:00 smithi002 ceph-mon[31328]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:00.823 DEBUG:teuthology.orchestra.run.smithi165:osd.8> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.8.service 2024-03-11T18:28:00.825 INFO:tasks.cephadm:Deploying osd.9 on smithi002 with /dev/vg_nvme/lv_1... 2024-03-11T18:28:00.826 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:00 smithi023 ceph-mon[36406]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-11T18:28:01.316 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:01 smithi002 ceph-mon[31328]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:01.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:01.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T18:28:01.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T18:28:01.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:01 smithi023 ceph-mon[36406]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: osdmap e59: 9 total, 8 up, 9 in 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: from='osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:01 smithi165 ceph-mon[36634]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:28:02.655 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:28:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:28:02.175+0000 7fccc8fa2700 -1 osd.8 0 waiting for initial osdmap 2024-03-11T18:28:02.655 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:28:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:28:02.188+0000 7fccc4d95700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:28:03.274 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:03.274 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:28:03.274 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T18:28:03.274 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] boot 2024-03-11T18:28:03.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T18:28:03.275 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:03.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:28:03.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T18:28:03.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] boot 2024-03-11T18:28:03.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T18:28:03.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: osdmap e60: 9 total, 8 up, 9 in 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: osd.8 [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] boot 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: osdmap e61: 9 total, 9 up, 9 in 2024-03-11T18:28:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10+0 records in 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr:10+0 records out 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0378848 s, 277 MB/s 2024-03-11T18:28:04.303 INFO:teuthology.orchestra.run.smithi002.stderr:--> Zapping successful for: 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi165 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 5693M 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.304 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:04.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:04.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:04 smithi002 ceph-mon[31328]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi165 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 5693M 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:04 smithi165 ceph-mon[36634]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T18:28:04.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi165 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 5693M 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:04 smithi023 ceph-mon[36406]: osdmap e62: 9 total, 9 up, 9 in 2024-03-11T18:28:05.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2024-03-11T18:28:05.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:05 smithi002 ceph-mon[31328]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:05.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:05 smithi002 ceph-mon[31328]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T18:28:05.634 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:28:05.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:05 smithi165 ceph-mon[36634]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:05.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:05 smithi165 ceph-mon[36634]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T18:28:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:05 smithi023 ceph-mon[36406]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:05 smithi023 ceph-mon[36406]: osdmap e63: 9 total, 9 up, 9 in 2024-03-11T18:28:07.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:07 smithi002 ceph-mon[31328]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:07.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:07 smithi002 ceph-mon[31328]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T18:28:07.457 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:07 smithi165 ceph-mon[36634]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:07.457 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:07 smithi165 ceph-mon[36634]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T18:28:07.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:07 smithi023 ceph-mon[36406]: pgmap v222: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:07.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:07 smithi023 ceph-mon[36406]: osdmap e64: 9 total, 9 up, 9 in 2024-03-11T18:28:07.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 -- 172.21.15.2:0/203351513 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4103e60 msgr2=0x7fb7e4104280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:07.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 --2- 172.21.15.2:0/203351513 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4103e60 0x7fb7e4104280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009990 tx=0x7fb7d4009ca0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 -- 172.21.15.2:0/203351513 shutdown_connections 2024-03-11T18:28:07.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 --2- 172.21.15.2:0/203351513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4105a10 0x7fb7e410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 --2- 172.21.15.2:0/203351513 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4103e60 0x7fb7e4104280 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 --2- 172.21.15.2:0/203351513 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105050 0x7fb7e41054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 -- 172.21.15.2:0/203351513 >> 172.21.15.2:0/203351513 conn(0x7fb7e40ff430 msgr2=0x7fb7e4101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:28:07.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 -- 172.21.15.2:0/203351513 shutdown_connections 2024-03-11T18:28:07.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.835+0000 7fb7eb200700 1 -- 172.21.15.2:0/203351513 wait complete. 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.836+0000 7fb7eb200700 1 Processor -- start 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.836+0000 7fb7eb200700 1 -- start start 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 0x7fb7e4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 0x7fb7e419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105a10 0x7fb7e41a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e41a19f0 con 0x7fb7e4105050 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e41a1b30 con 0x7fb7e4103e60 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7eb200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e41a1cc0 con 0x7fb7e4105a10 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7dbe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 0x7fb7e419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:07.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7dbe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 0x7fb7e419a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39872/0 (socket says 172.21.15.2:39872) 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7dbe00700 1 -- 172.21.15.2:0/450918388 learned_addr learned my addr 172.21.15.2:0/450918388 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 0x7fb7e4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e9200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105a10 0x7fb7e41a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 -- 172.21.15.2:0/450918388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105a10 msgr2=0x7fb7e41a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105a10 0x7fb7e41a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 -- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 msgr2=0x7fb7e419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 0x7fb7e419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.837+0000 7fb7e3e00700 1 -- 172.21.15.2:0/450918388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc009c40 con 0x7fb7e4103e60 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7e3e00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 0x7fb7e4199e60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400b690 tx=0x7fb7d4009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d401d070 con 0x7fb7e4103e60 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d40095f0 con 0x7fb7e4103e60 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d40047e0 con 0x7fb7e4103e60 2024-03-11T18:28:07.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4005430 con 0x7fb7e4103e60 2024-03-11T18:28:07.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.838+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e41a21c0 con 0x7fb7e4103e60 2024-03-11T18:28:07.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.839+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e410b240 con 0x7fb7e4103e60 2024-03-11T18:28:07.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.841+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fb7d4005590 con 0x7fb7e4103e60 2024-03-11T18:28:07.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.843+0000 7fb7e1600700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 0x7fb7c80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:07.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.843+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5829+0+0 (secure 0 0 0) 0x7fb7d40cdca0 con 0x7fb7e4103e60 2024-03-11T18:28:07.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.843+0000 7fb7dbe00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 0x7fb7c80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:07.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.844+0000 7fb7dbe00700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 0x7fb7c80a3f80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00bd70 tx=0x7fb7cc00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:28:07.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:07.845+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb7d4097850 con 0x7fb7e4103e60 2024-03-11T18:28:08.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:08.085+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7e402d8d0 con 0x7fb7c80a1ac0 2024-03-11T18:28:08.470 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:08 smithi165 ceph-mon[36634]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T18:28:08.470 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:08 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:08.471 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:08 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:08.471 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:08 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:08 smithi002 ceph-mon[31328]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T18:28:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:08 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:08 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:08 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:08 smithi023 ceph-mon[36406]: osdmap e65: 9 total, 9 up, 9 in 2024-03-11T18:28:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:08 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:08 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:08 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:09 smithi002 ceph-mon[31328]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:09 smithi002 ceph-mon[31328]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:28:09.647 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:09 smithi165 ceph-mon[36634]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:09.647 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:09 smithi165 ceph-mon[36634]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:28:09.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:09 smithi023 ceph-mon[36406]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:09.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:09 smithi023 ceph-mon[36406]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:28:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:11 smithi002 ceph-mon[31328]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:11 smithi165 ceph-mon[36634]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:11.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:11 smithi023 ceph-mon[36406]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 262 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:12 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]: dispatch 2024-03-11T18:28:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:12 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]': finished 2024-03-11T18:28:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:12 smithi002 ceph-mon[31328]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T18:28:12.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:12 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:12 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]: dispatch 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:12 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]': finished 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:12 smithi165 ceph-mon[36634]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:12 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:12 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]: dispatch 2024-03-11T18:28:12.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:12 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1608393304' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1ca1d014-c271-49e5-9672-4ff1457beb85"}]': finished 2024-03-11T18:28:12.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:12 smithi023 ceph-mon[36406]: osdmap e66: 10 total, 9 up, 10 in 2024-03-11T18:28:12.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:12 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:13.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:13 smithi002 ceph-mon[31328]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:13.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:13 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1359786431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:13.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:13 smithi165 ceph-mon[36634]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:13.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:13 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1359786431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:13.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:13 smithi023 ceph-mon[36406]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T18:28:13.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:13 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1359786431' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:15.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:15 smithi002 ceph-mon[31328]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:28:15.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:15 smithi165 ceph-mon[36634]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:28:15.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:15 smithi023 ceph-mon[36406]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T18:28:17.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:17 smithi002 ceph-mon[31328]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:28:17.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:17 smithi165 ceph-mon[36634]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:28:17.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:17 smithi023 ceph-mon[36406]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-11T18:28:19.443 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:19 smithi002 ceph-mon[31328]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:19 smithi023 ceph-mon[36406]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:19 smithi165 ceph-mon[36634]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:21.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:21 smithi002 ceph-mon[31328]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:21.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:21 smithi165 ceph-mon[36634]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:21.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:21 smithi023 ceph-mon[36406]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T18:28:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:22 smithi002 ceph-mon[31328]: Deploying daemon osd.9 on smithi002 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:22 smithi165 ceph-mon[36634]: Deploying daemon osd.9 on smithi002 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:22.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:22 smithi023 ceph-mon[36406]: Deploying daemon osd.9 on smithi002 2024-03-11T18:28:23.487 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:23 smithi002 ceph-mon[31328]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:23.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:23 smithi165 ceph-mon[36634]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:23 smithi023 ceph-mon[36406]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:25.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:25 smithi002 ceph-mon[31328]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:25.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:25 smithi165 ceph-mon[36634]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:25.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:25 smithi023 ceph-mon[36406]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:27 smithi165 ceph-mon[36634]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:27.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:27 smithi023 ceph-mon[36406]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:27.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:27 smithi002 ceph-mon[31328]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:29.522 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:29 smithi002 ceph-mon[31328]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:29.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:29 smithi165 ceph-mon[36634]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:29 smithi023 ceph-mon[36406]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:31.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:31.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:31 smithi002 ceph-mon[31328]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:31 smithi165 ceph-mon[36634]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:32.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:31 smithi023 ceph-mon[36406]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:34.182 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:33 smithi002 ceph-mon[31328]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:34.182 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:33 smithi002 ceph-mon[31328]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:34.182 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:33 smithi002 ceph-mon[31328]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:34.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.283+0000 7fb7e1600700 1 -- 172.21.15.2:0/450918388 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb7e402d8d0 con 0x7fb7c80a1ac0 2024-03-11T18:28:34.286 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 9 on host 'smithi002' 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 msgr2=0x7fb7c80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 0x7fb7c80a3f80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00bd70 tx=0x7fb7cc00b4a0 comp rx=0 tx=0).stop 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 msgr2=0x7fb7e4199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 0x7fb7e4199e60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400b690 tx=0x7fb7d4009580 comp rx=0 tx=0).stop 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 shutdown_connections 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb7c80a1ac0 0x7fb7c80a3f80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7e4105a10 0x7fb7e41a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4103e60 0x7fb7e4199e60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 --2- 172.21.15.2:0/450918388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7e4105050 0x7fb7e419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.286+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 >> 172.21.15.2:0/450918388 conn(0x7fb7e40ff430 msgr2=0x7fb7e4108380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:28:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.287+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 shutdown_connections 2024-03-11T18:28:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:28:34.287+0000 7fb7eb200700 1 -- 172.21.15.2:0/450918388 wait complete. 2024-03-11T18:28:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:33 smithi023 ceph-mon[36406]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:33 smithi023 ceph-mon[36406]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:33 smithi023 ceph-mon[36406]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:33 smithi165 ceph-mon[36634]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:33 smithi165 ceph-mon[36634]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:34.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:33 smithi165 ceph-mon[36634]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-11T18:28:35.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T18:28:35.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:34 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:35.121 DEBUG:teuthology.orchestra.run.smithi002:osd.9> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.9.service 2024-03-11T18:28:35.128 INFO:tasks.cephadm:Deploying osd.10 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-11T18:28:35.129 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T18:28:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T18:28:35.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:34 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: osdmap e67: 10 total, 9 up, 10 in 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:35.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-03-11T18:28:35.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: osdmap e68: 10 total, 9 up, 10 in 2024-03-11T18:28:35.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:34 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:36.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:35 smithi002 ceph-mon[31328]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:36.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:35 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:36.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:35 smithi002 ceph-mon[31328]: from='osd.9 ' entity='osd.9' 2024-03-11T18:28:36.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:28:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:28:35.727+0000 7f35ae400700 -1 osd.9 0 waiting for initial osdmap 2024-03-11T18:28:36.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:28:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:28:35.768+0000 7f35a4200700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:28:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:35 smithi023 ceph-mon[36406]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:35 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:35 smithi023 ceph-mon[36406]: from='osd.9 ' entity='osd.9' 2024-03-11T18:28:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:35 smithi165 ceph-mon[36634]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:35 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:35 smithi165 ceph-mon[36634]: from='osd.9 ' entity='osd.9' 2024-03-11T18:28:37.226 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:28:37.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:28:37.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] boot 2024-03-11T18:28:37.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T18:28:37.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:36 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] boot 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T18:28:37.292 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:36 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: osd.9 [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] boot 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: osdmap e69: 10 total, 10 up, 10 in 2024-03-11T18:28:37.306 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:36 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00703968 s, 1.5 GB/s 2024-03-11T18:28:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-11T18:28:38.086 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:37 smithi023 ceph-mon[36406]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:38.086 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:37 smithi023 ceph-mon[36406]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T18:28:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:37 smithi002 ceph-mon[31328]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:37 smithi002 ceph-mon[31328]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T18:28:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:37 smithi165 ceph-mon[36634]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-11T18:28:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:37 smithi165 ceph-mon[36634]: osdmap e70: 10 total, 10 up, 10 in 2024-03-11T18:28:38.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-11T18:28:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:28:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:39 smithi002 ceph-mon[31328]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:40.258 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:39 smithi023 ceph-mon[36406]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:40.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:39 smithi165 ceph-mon[36634]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.555+0000 7f43678e3700 1 -- 172.21.15.23:0/1132849386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4360107a30 msgr2=0x7f4360109ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.555+0000 7f43678e3700 1 --2- 172.21.15.23:0/1132849386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4360107a30 0x7f4360109ea0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f435800b0a0 tx=0x7f435800b3b0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- 172.21.15.23:0/1132849386 shutdown_connections 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- 172.21.15.23:0/1132849386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4360107a30 0x7f4360109ea0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- 172.21.15.23:0/1132849386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43601027d0 0x7f4360104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- 172.21.15.23:0/1132849386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f43601074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- 172.21.15.23:0/1132849386 >> 172.21.15.23:0/1132849386 conn(0x7f43600fc200 msgr2=0x7f43600fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- 172.21.15.23:0/1132849386 shutdown_connections 2024-03-11T18:28:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- 172.21.15.23:0/1132849386 wait complete. 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 Processor -- start 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- start start 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f436019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 0x7f43601a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43601a1b90 con 0x7f4360107a30 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43601a1cd0 con 0x7f43601027d0 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.556+0000 7f43678e3700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43601a1e60 con 0x7f4360105100 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38416/0 (socket says 172.21.15.23:38416) 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 -- 172.21.15.23:0/1708892153 learned_addr learned my addr 172.21.15.23:0/1708892153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-11T18:28:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f4364e7e700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f436019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 -- 172.21.15.23:0/1708892153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 msgr2=0x7f436019a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f436019a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f4365e80700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 0x7f43601a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 -- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 msgr2=0x7f43601a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 0x7f43601a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 -- 172.21.15.23:0/1708892153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4358009d00 con 0x7f43601027d0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f4364e7e700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f436019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f4365e80700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 0x7f43601a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f436567f700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f435c00b530 tx=0x7f435c00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c011830 con 0x7f43601027d0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f435c011e70 con 0x7f43601027d0 2024-03-11T18:28:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43601a2140 con 0x7f43601027d0 2024-03-11T18:28:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00f2e0 con 0x7f43601027d0 2024-03-11T18:28:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.557+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43601a26c0 con 0x7f43601027d0 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.558+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43601095b0 con 0x7f43601027d0 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.559+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f435c0105f0 con 0x7f43601027d0 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.560+0000 7f43567fc700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 0x7f434c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.560+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f435c0cc730 con 0x7f43601027d0 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.560+0000 7f4364e7e700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 0x7f434c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:28:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.561+0000 7f4364e7e700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 0x7f434c0a3f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4350005950 tx=0x7f4350005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:28:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.564+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f435c096140 con 0x7f43601027d0 2024-03-11T18:28:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:28:40.780+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f436002d8d0 con 0x7f434c0a1ac0 2024-03-11T18:28:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:40 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:40 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:28:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:40 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:42.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:41 smithi023 ceph-mon[36406]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:42.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:41 smithi023 ceph-mon[36406]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:41 smithi002 ceph-mon[31328]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:41 smithi002 ceph-mon[31328]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:41 smithi165 ceph-mon[36634]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:41 smithi165 ceph-mon[36634]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:28:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi002 2024-03-11T18:28:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 4273M 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:43 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi002 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 4273M 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:43.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:43 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi002 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 4273M 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:43.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:43.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:43.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:28:43.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:43 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:44 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/239727719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:44 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:44 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]': finished 2024-03-11T18:28:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:44 smithi023 ceph-mon[36406]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T18:28:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:44 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:28:44.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/239727719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:44 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:44 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]': finished 2024-03-11T18:28:44.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:44 smithi002 ceph-mon[31328]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T18:28:44.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:44 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:28:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:44 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/239727719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:44 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]: dispatch 2024-03-11T18:28:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:44 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54"}]': finished 2024-03-11T18:28:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:44 smithi165 ceph-mon[36634]: osdmap e71: 11 total, 10 up, 11 in 2024-03-11T18:28:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:44 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:28:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:45 smithi002 ceph-mon[31328]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.23:0/3329433656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:45.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:45 smithi165 ceph-mon[36634]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:45.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:45 smithi165 ceph-mon[36634]: from='client.? 172.21.15.23:0/3329433656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:45.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:45 smithi023 ceph-mon[36406]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:45.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:45 smithi023 ceph-mon[36406]: from='client.? 172.21.15.23:0/3329433656' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:28:47.431 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:47 smithi023 ceph-mon[36406]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:47.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:47 smithi002 ceph-mon[31328]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:47.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:47 smithi165 ceph-mon[36634]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:49 smithi023 ceph-mon[36406]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:49 smithi002 ceph-mon[31328]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:49.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:49 smithi165 ceph-mon[36634]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:51.376 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:51 smithi023 ceph-mon[36406]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:51.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:51 smithi002 ceph-mon[31328]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:51.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:51 smithi165 ceph-mon[36634]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:52.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T18:28:52.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:52 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:52.338 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:52 smithi023 ceph-mon[36406]: Deploying daemon osd.10 on smithi023 2024-03-11T18:28:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T18:28:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:52 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:52 smithi002 ceph-mon[31328]: Deploying daemon osd.10 on smithi023 2024-03-11T18:28:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-11T18:28:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:52 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:28:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:52 smithi165 ceph-mon[36634]: Deploying daemon osd.10 on smithi023 2024-03-11T18:28:53.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:53 smithi023 ceph-mon[36406]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:53.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:53 smithi002 ceph-mon[31328]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:53 smithi165 ceph-mon[36634]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:55.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:55 smithi023 ceph-mon[36406]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:55.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:55 smithi002 ceph-mon[31328]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:55.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:55 smithi165 ceph-mon[36634]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:57.456 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:57 smithi023 ceph-mon[36406]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:57.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:57 smithi002 ceph-mon[31328]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:57.654 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:57 smithi165 ceph-mon[36634]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:58 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:58 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:59.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:59.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:28:59.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:58 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:28:59.896 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:28:59 smithi023 ceph-mon[36406]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:28:59.904 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:28:59 smithi165 ceph-mon[36634]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:00.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:28:59 smithi002 ceph-mon[31328]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:01.569 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 10 on host 'smithi023' 2024-03-11T18:29:01.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.567+0000 7f43567fc700 1 -- 172.21.15.23:0/1708892153 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f436002d8d0 con 0x7f434c0a1ac0 2024-03-11T18:29:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 msgr2=0x7f434c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 0x7f434c0a3f80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4350005950 tx=0x7f4350005f50 comp rx=0 tx=0).stop 2024-03-11T18:29:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 msgr2=0x7f436019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f435c00b530 tx=0x7f435c00b8f0 comp rx=0 tx=0).stop 2024-03-11T18:29:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 shutdown_connections 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4360107a30 0x7f43601a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601027d0 0x7f436019a000 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f434c0a1ac0 0x7f434c0a3f80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 --2- 172.21.15.23:0/1708892153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4360105100 0x7f436019a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 >> 172.21.15.23:0/1708892153 conn(0x7f43600fc200 msgr2=0x7f4360101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 shutdown_connections 2024-03-11T18:29:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-11T18:29:01.570+0000 7f43678e3700 1 -- 172.21.15.23:0/1708892153 wait complete. 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:01 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:01.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:01 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-11T18:29:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:01 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.332 DEBUG:teuthology.orchestra.run.smithi023:osd.10> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.10.service 2024-03-11T18:29:02.334 INFO:tasks.cephadm:Deploying osd.11 on smithi165 with /dev/vg_nvme/lv_1... 2024-03-11T18:29:02.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T18:29:02.583 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:02 smithi165 ceph-mon[36634]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T18:29:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:02.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T18:29:02.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:02.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:02 smithi023 ceph-mon[36406]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:03.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:03.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-11T18:29:03.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: from='osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:03.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: osdmap e72: 11 total, 10 up, 11 in 2024-03-11T18:29:03.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:03.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:02 smithi002 ceph-mon[31328]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-11T18:29:03.814 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-mon[36406]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:03.814 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-mon[36406]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:29:03.814 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-mon[36406]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T18:29:03.814 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:03.814 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:03.815 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:29:03.528+0000 7f1766726700 -1 osd.10 0 waiting for initial osdmap 2024-03-11T18:29:03.815 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:29:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:29:03.542+0000 7f1762519700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:29:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:03 smithi165 ceph-mon[36634]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:03 smithi165 ceph-mon[36634]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:29:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:03 smithi165 ceph-mon[36634]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T18:29:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:04.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:03 smithi002 ceph-mon[31328]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:04.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:03 smithi002 ceph-mon[31328]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-11T18:29:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:03 smithi002 ceph-mon[31328]: osdmap e73: 11 total, 10 up, 11 in 2024-03-11T18:29:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113248 s, 926 MB/s 2024-03-11T18:29:04.641 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-03-11T18:29:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:04 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:29:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:04 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:29:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:04 smithi165 ceph-mon[36634]: osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] boot 2024-03-11T18:29:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:04 smithi165 ceph-mon[36634]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T18:29:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:04 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:05.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:04 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:29:05.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:04 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:29:05.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:04 smithi002 ceph-mon[31328]: osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] boot 2024-03-11T18:29:05.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:04 smithi002 ceph-mon[31328]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T18:29:05.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:04 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:05.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:04 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:29:05.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:04 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:29:05.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:04 smithi023 ceph-mon[36406]: osd.10 [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] boot 2024-03-11T18:29:05.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:04 smithi023 ceph-mon[36406]: osdmap e74: 11 total, 11 up, 11 in 2024-03-11T18:29:05.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:04 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:29:05.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-03-11T18:29:05.524 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:29:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T18:29:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:05.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: osdmap e75: 11 total, 11 up, 11 in 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi023 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 4270M 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:06.887 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:06 smithi165 ceph-mon[36634]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T18:29:07.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi023 2024-03-11T18:29:07.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 4270M 2024-03-11T18:29:07.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:07.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:07.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:07.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:07.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:06 smithi002 ceph-mon[31328]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 -- 172.21.15.165:0/3367594672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510102e90 msgr2=0x7f25101032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 --2- 172.21.15.165:0/3367594672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510102e90 0x7f25101032f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f250c009990 tx=0x7f250c009ca0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 -- 172.21.15.165:0/3367594672 shutdown_connections 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 --2- 172.21.15.165:0/3367594672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510103830 0x7f251010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 --2- 172.21.15.165:0/3367594672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2510101c90 0x7f25101020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 --2- 172.21.15.165:0/3367594672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510102e90 0x7f25101032f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.088+0000 7f2517f8b700 1 -- 172.21.15.165:0/3367594672 >> 172.21.15.165:0/3367594672 conn(0x7f25100fd250 msgr2=0x7f25100ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 -- 172.21.15.165:0/3367594672 shutdown_connections 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 -- 172.21.15.165:0/3367594672 wait complete. 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 Processor -- start 2024-03-11T18:29:07.091 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 -- start start 2024-03-11T18:29:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2510101c90 0x7f2510197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510102e90 0x7f25101982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.089+0000 7f2517f8b700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251019f930 con 0x7f2510101c90 2024-03-11T18:29:07.093 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2517f8b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251019fa70 con 0x7f2510102e90 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2517f8b700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251019fc00 con 0x7f2510103830 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:37948/0 (socket says 172.21.15.165:37948) 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 -- 172.21.15.165:0/1819302412 learned_addr learned my addr 172.21.15.165:0/1819302412 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2515526700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510102e90 0x7f25101982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 -- 172.21.15.165:0/1819302412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510102e90 msgr2=0x7f25101982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510102e90 0x7f25101982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 -- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2510101c90 msgr2=0x7f2510197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2510101c90 0x7f2510197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 -- 172.21.15.165:0/1819302412 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c0095f0 con 0x7f2510103830 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2516528700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f250800b9a0 tx=0x7f250800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250800c9b0 con 0x7f2510103830 2024-03-11T18:29:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2508014440 con 0x7f2510103830 2024-03-11T18:29:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f251019fee0 con 0x7f2510103830 2024-03-11T18:29:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25080075d0 con 0x7f2510103830 2024-03-11T18:29:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.090+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25101a0380 con 0x7f2510103830 2024-03-11T18:29:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.091+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2510109110 con 0x7f2510103830 2024-03-11T18:29:07.096 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.092+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f2508014920 con 0x7f2510103830 2024-03-11T18:29:07.096 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.093+0000 7f2506ffd700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 0x7f24fc0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:07.096 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.093+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f2508006e40 con 0x7f2510103830 2024-03-11T18:29:07.096 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.094+0000 7f2515d27700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 0x7f24fc0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:07.096 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.095+0000 7f2515d27700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 0x7f24fc0a3f80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2510102cf0 tx=0x7f2500006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:07.098 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.097+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2508096230 con 0x7f2510103830 2024-03-11T18:29:07.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi023 2024-03-11T18:29:07.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 4270M 2024-03-11T18:29:07.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:07.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:07.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:07.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:07.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:06 smithi023 ceph-mon[36406]: osdmap e76: 11 total, 11 up, 11 in 2024-03-11T18:29:07.304 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:07.303+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f251002d920 con 0x7f24fc0a1ac0 2024-03-11T18:29:08.249 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:07 smithi165 ceph-mon[36634]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:08.249 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:29:08.249 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:29:08.249 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:07 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:08.249 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:07 smithi165 ceph-mon[36634]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T18:29:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:07 smithi002 ceph-mon[31328]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:29:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:29:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:07 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:07 smithi002 ceph-mon[31328]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T18:29:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:07 smithi023 ceph-mon[36406]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T18:29:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T18:29:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:07 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:07 smithi023 ceph-mon[36406]: osdmap e77: 11 total, 11 up, 11 in 2024-03-11T18:29:09.213 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:08 smithi165 ceph-mon[36634]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:09.213 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:08 smithi165 ceph-mon[36634]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T18:29:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:08 smithi002 ceph-mon[31328]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:09.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:08 smithi002 ceph-mon[31328]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T18:29:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:08 smithi023 ceph-mon[36406]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:08 smithi023 ceph-mon[36406]: osdmap e78: 11 total, 11 up, 11 in 2024-03-11T18:29:10.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:09 smithi002 ceph-mon[31328]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:10.392 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:09 smithi165 ceph-mon[36634]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:10.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:09 smithi023 ceph-mon[36406]: pgmap v267: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:11.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/2067797060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]': finished 2024-03-11T18:29:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T18:29:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:11.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:10 smithi002 ceph-mon[31328]: from='client.? 172.21.15.165:0/2159934495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:29:11.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/2067797060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]': finished 2024-03-11T18:29:11.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T18:29:11.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:11.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:10 smithi165 ceph-mon[36634]: from='client.? 172.21.15.165:0/2159934495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:29:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/2067797060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]: dispatch 2024-03-11T18:29:11.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c89611-028f-40d8-a5b0-d36c76cd8800"}]': finished 2024-03-11T18:29:11.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: osdmap e79: 12 total, 11 up, 12 in 2024-03-11T18:29:11.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:11.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:10 smithi023 ceph-mon[36406]: from='client.? 172.21.15.165:0/2159934495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T18:29:12.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:12 smithi165 ceph-mon[36634]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:12 smithi023 ceph-mon[36406]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:12 smithi002 ceph-mon[31328]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T18:29:14.295 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:14 smithi165 ceph-mon[36634]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-11T18:29:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:14 smithi023 ceph-mon[36406]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-11T18:29:14.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:14 smithi002 ceph-mon[31328]: pgmap v271: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-11T18:29:16.256 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:16 smithi165 ceph-mon[36634]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T18:29:16.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:16 smithi023 ceph-mon[36406]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T18:29:16.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:16 smithi002 ceph-mon[31328]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T18:29:18.363 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:18 smithi165 ceph-mon[36634]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:18.363 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T18:29:18.363 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:18 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:18 smithi023 ceph-mon[36406]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T18:29:18.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:18 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:18 smithi002 ceph-mon[31328]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-11T18:29:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:18 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:19.259 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:19 smithi165 ceph-mon[36634]: Deploying daemon osd.11 on smithi165 2024-03-11T18:29:19.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:19 smithi023 ceph-mon[36406]: Deploying daemon osd.11 on smithi165 2024-03-11T18:29:19.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:19 smithi002 ceph-mon[31328]: Deploying daemon osd.11 on smithi165 2024-03-11T18:29:20.399 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:20 smithi165 ceph-mon[36634]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:20.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:20 smithi023 ceph-mon[36406]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:20 smithi002 ceph-mon[31328]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:21.337 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:21 smithi165 ceph-mon[36634]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:21.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:21 smithi023 ceph-mon[36406]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:21.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:21 smithi002 ceph-mon[31328]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:23 smithi002 ceph-mon[31328]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:23.510 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:23 smithi165 ceph-mon[36634]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:23.510 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.510 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.510 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:23 smithi023 ceph-mon[36406]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:23.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:25.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:25 smithi165 ceph-mon[36634]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:25.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:25 smithi002 ceph-mon[31328]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:25.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:25 smithi023 ceph-mon[36406]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:26.844 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 11 on host 'smithi165' 2024-03-11T18:29:26.844 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.842+0000 7f2506ffd700 1 -- 172.21.15.165:0/1819302412 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f251002d920 con 0x7f24fc0a1ac0 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 msgr2=0x7f24fc0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 0x7f24fc0a3f80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2510102cf0 tx=0x7f2500006d20 comp rx=0 tx=0).stop 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 msgr2=0x7f251019f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f250800b9a0 tx=0x7f250800bd60 comp rx=0 tx=0).stop 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 shutdown_connections 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24fc0a1ac0 0x7f24fc0a3f80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:26.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2510103830 0x7f251019f360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:26.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2510101c90 0x7f2510197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:26.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 --2- 172.21.15.165:0/1819302412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2510102e90 0x7f25101982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:26.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 >> 172.21.15.165:0/1819302412 conn(0x7f25100fd250 msgr2=0x7f2510106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:26.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 shutdown_connections 2024-03-11T18:29:26.847 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:26.844+0000 7f2517f8b700 1 -- 172.21.15.165:0/1819302412 wait complete. 2024-03-11T18:29:27.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:27 smithi023 ceph-mon[36406]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:27 smithi165 ceph-mon[36634]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.493 DEBUG:teuthology.orchestra.run.smithi165:osd.11> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.11.service 2024-03-11T18:29:27.496 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-11T18:29:27.496 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T18:29:27.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:27 smithi002 ceph-mon[31328]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-11T18:29:27.969 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-mon[36634]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:28.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T18:29:28.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T18:29:28.339 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:28.340 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:28 smithi002 ceph-mon[31328]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: from='osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: osdmap e80: 12 total, 11 up, 12 in 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:28.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:28 smithi023 ceph-mon[36406]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-03-11T18:29:28.977 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:29:28.815+0000 7f6ae11eb700 -1 osd.11 0 waiting for initial osdmap 2024-03-11T18:29:28.977 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:29:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:29:28.828+0000 7f6adcfde700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T18:29:30.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:29 smithi002 ceph-mon[31328]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:30.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:29 smithi002 ceph-mon[31328]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:29:30.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:29 smithi002 ceph-mon[31328]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T18:29:30.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:30.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:29 smithi165 ceph-mon[36634]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:30.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:29 smithi165 ceph-mon[36634]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:29:30.082 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:29 smithi165 ceph-mon[36634]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T18:29:30.083 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:29 smithi023 ceph-mon[36406]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-11T18:29:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:29 smithi023 ceph-mon[36406]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-03-11T18:29:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:29 smithi023 ceph-mon[36406]: osdmap e81: 12 total, 11 up, 12 in 2024-03-11T18:29:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:30.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 -- 172.21.15.2:0/845040015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c074670 msgr2=0x7fe01c074af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 --2- 172.21.15.2:0/845040015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c074670 0x7fe01c074af0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe008009990 tx=0x7fe008009ca0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 -- 172.21.15.2:0/845040015 shutdown_connections 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 --2- 172.21.15.2:0/845040015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c109bd0 0x7fe01c10c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 --2- 172.21.15.2:0/845040015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c075c40 0x7fe01c0740a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 --2- 172.21.15.2:0/845040015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c074670 0x7fe01c074af0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 -- 172.21.15.2:0/845040015 >> 172.21.15.2:0/845040015 conn(0x7fe01c0fd290 msgr2=0x7fe01c0ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 -- 172.21.15.2:0/845040015 shutdown_connections 2024-03-11T18:29:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.347+0000 7fe023c00700 1 -- 172.21.15.2:0/845040015 wait complete. 2024-03-11T18:29:30.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 Processor -- start 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 -- start start 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c075c40 0x7fe01c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c109bd0 0x7fe01c1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c1a1950 con 0x7fe01c075c40 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c1a1a90 con 0x7fe01c109bd0 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.349+0000 7fe023c00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c1a1c20 con 0x7fe01c074670 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:43564/0 (socket says 172.21.15.2:43564) 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe01be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c075c40 0x7fe01c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 -- 172.21.15.2:0/2723307302 learned_addr learned my addr 172.21.15.2:0/2723307302 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c109bd0 0x7fe01c1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 -- 172.21.15.2:0/2723307302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c109bd0 msgr2=0x7fe01c1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c109bd0 0x7fe01c1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 -- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c075c40 msgr2=0x7fe01c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c075c40 0x7fe01c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 -- 172.21.15.2:0/2723307302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010009520 con 0x7fe01c074670 2024-03-11T18:29:30.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.350+0000 7fe021200700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe01000eb60 tx=0x7fe01000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.351+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000cc90 con 0x7fe01c074670 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.351+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe010004500 con 0x7fe01c074670 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.351+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010010430 con 0x7fe01c074670 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.351+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0080095f0 con 0x7fe01c074670 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.351+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c1a21c0 con 0x7fe01c074670 2024-03-11T18:29:30.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.352+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c068970 con 0x7fe01c074670 2024-03-11T18:29:30.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.354+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fe010003710 con 0x7fe01c074670 2024-03-11T18:29:30.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.356+0000 7fe019600700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 0x7fe0040a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:30.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.356+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe010014070 con 0x7fe01c074670 2024-03-11T18:29:30.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.356+0000 7fe01be00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 0x7fe0040a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:30.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.357+0000 7fe01be00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 0x7fe0040a4000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe00800b650 tx=0x7fe008005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:30.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.359+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe010096920 con 0x7fe01c074670 2024-03-11T18:29:30.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.596+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe01c10f810 con 0x7fe01c074670 2024-03-11T18:29:30.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.596+0000 7fe019600700 1 -- 172.21.15.2:0/2723307302 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe010099fe0 con 0x7fe01c074670 2024-03-11T18:29:30.598 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 msgr2=0x7fe0040a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 0x7fe0040a4000 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe00800b650 tx=0x7fe008005fb0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 msgr2=0x7fe01c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:30.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe01000eb60 tx=0x7fe01000ee70 comp rx=0 tx=0).stop 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 shutdown_connections 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0040a1b40 0x7fe0040a4000 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe01c074670 0x7fe01c199dc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe01c109bd0 0x7fe01c1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 --2- 172.21.15.2:0/2723307302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe01c075c40 0x7fe01c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 >> 172.21.15.2:0/2723307302 conn(0x7fe01c0fd290 msgr2=0x7fe01c105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:30.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 shutdown_connections 2024-03-11T18:29:30.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:30.599+0000 7fe023c00700 1 -- 172.21.15.2:0/2723307302 wait complete. 2024-03-11T18:29:30.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: purged_snaps scrub starts 2024-03-11T18:29:30.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: purged_snaps scrub ok 2024-03-11T18:29:30.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] boot 2024-03-11T18:29:30.998 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:30.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:30 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2723307302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: purged_snaps scrub starts 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: purged_snaps scrub ok 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] boot 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:30 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2723307302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: purged_snaps scrub starts 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: purged_snaps scrub ok 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: osd.11 [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] boot 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: osdmap e82: 12 total, 12 up, 12 in 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:29:31.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:31.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:30 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2723307302' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T18:29:31.391 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1710181769,"num_in_osds":12,"osd_in_since":1710181750,"num_remapped_pgs":0} 2024-03-11T18:29:31.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T18:29:31.822 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:29:31.966 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:31 smithi002 ceph-mon[31328]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:31.966 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:31 smithi002 ceph-mon[31328]: Detected new or changed devices on smithi165 2024-03-11T18:29:31.966 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:31 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 4270M 2024-03-11T18:29:31.966 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:31 smithi002 ceph-mon[31328]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T18:29:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:31 smithi165 ceph-mon[36634]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:31 smithi165 ceph-mon[36634]: Detected new or changed devices on smithi165 2024-03-11T18:29:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:31 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 4270M 2024-03-11T18:29:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:31 smithi165 ceph-mon[36634]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T18:29:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:31 smithi023 ceph-mon[36406]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:31 smithi023 ceph-mon[36406]: Detected new or changed devices on smithi165 2024-03-11T18:29:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:31 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 4270M 2024-03-11T18:29:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:31 smithi023 ceph-mon[36406]: osdmap e83: 12 total, 12 up, 12 in 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 -- 172.21.15.2:0/3492250765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 msgr2=0x7f66f8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 --2- 172.21.15.2:0/3492250765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f8108ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009990 tx=0x7f66e4009ca0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 -- 172.21.15.2:0/3492250765 shutdown_connections 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 --2- 172.21.15.2:0/3492250765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f8110d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 --2- 172.21.15.2:0/3492250765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 0x7f66f81044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 --2- 172.21.15.2:0/3492250765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f8108ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 -- 172.21.15.2:0/3492250765 >> 172.21.15.2:0/3492250765 conn(0x7f66f80ff970 msgr2=0x7f66f8101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:33.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 -- 172.21.15.2:0/3492250765 shutdown_connections 2024-03-11T18:29:33.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.979+0000 7f66fd200700 1 -- 172.21.15.2:0/3492250765 wait complete. 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.980+0000 7f66fd200700 1 Processor -- start 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.980+0000 7f66fd200700 1 -- start start 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 0x7f66f819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f819f2a0 con 0x7f66f81040e0 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f819f410 con 0x7f66f8104a90 2024-03-11T18:29:33.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66fd200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f819f5d0 con 0x7f66f81091c0 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66f6800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66f5e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 0x7f66f819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66f6800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:43582/0 (socket says 172.21.15.2:43582) 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66f6800700 1 -- 172.21.15.2:0/2447264709 learned_addr learned my addr 172.21.15.2:0/2447264709 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.981+0000 7f66f5400700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 -- 172.21.15.2:0/2447264709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 msgr2=0x7f66f819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 -- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 msgr2=0x7f66f819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 0x7f66f819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 -- 172.21.15.2:0/2447264709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e8009520 con 0x7f66f81091c0 2024-03-11T18:29:33.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f5400700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66f6800700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66f000ef30 tx=0x7f66f000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0018070 con 0x7f66f81091c0 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66f000f040 con 0x7f66f81091c0 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.982+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f000cd50 con 0x7f66f81091c0 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.983+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e40095f0 con 0x7f66f81091c0 2024-03-11T18:29:33.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.983+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f81a6f40 con 0x7f66f81091c0 2024-03-11T18:29:33.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.984+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f810c430 con 0x7f66f81091c0 2024-03-11T18:29:33.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.986+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f66f0007500 con 0x7f66f81091c0 2024-03-11T18:29:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.987+0000 7f66eea00700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 0x7f66dc0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.987+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f66f00cca60 con 0x7f66f81091c0 2024-03-11T18:29:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.987+0000 7f66f5e00700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 0x7f66dc0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:33.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.988+0000 7f66f5e00700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 0x7f66dc0a4000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009df0 tx=0x7f66e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:33.991+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66f0096100 con 0x7f66f81091c0 2024-03-11T18:29:34.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:33 smithi165 ceph-mon[36634]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:34.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:33 smithi023 ceph-mon[36406]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:34.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.219+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66f8068930 con 0x7f66f81091c0 2024-03-11T18:29:34.221 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:33 smithi002 ceph-mon[31328]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:34.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.220+0000 7f66eea00700 1 -- 172.21.15.2:0/2447264709 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18375 (secure 0 0 0) 0x7f66f00997c0 con 0x7f66f81091c0 2024-03-11T18:29:34.223 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:34.223 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:29:30.822013+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:29:34.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 msgr2=0x7f66dc0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:34.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 0x7f66dc0a4000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009df0 tx=0x7f66e8009260 comp rx=0 tx=0).stop 2024-03-11T18:29:34.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 msgr2=0x7f66f819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66f000ef30 tx=0x7f66f000c3a0 comp rx=0 tx=0).stop 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 shutdown_connections 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f66dc0a1b40 0x7f66dc0a4000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f66f81091c0 0x7f66f819ec90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66f81040e0 0x7f66f819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 --2- 172.21.15.2:0/2447264709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66f8104a90 0x7f66f819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 >> 172.21.15.2:0/2447264709 conn(0x7f66f80ff970 msgr2=0x7f66f8100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 shutdown_connections 2024-03-11T18:29:34.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:34.224+0000 7f66fd200700 1 -- 172.21.15.2:0/2447264709 wait complete. 2024-03-11T18:29:35.011 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T18:29:35.012 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T18:29:35.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:34 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2447264709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:35.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:34 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2447264709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:35.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:34 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2447264709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:35.484 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:29:35.996 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:35 smithi002 ceph-mon[31328]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:36.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:35 smithi165 ceph-mon[36634]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:35 smithi023 ceph-mon[36406]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:37.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 -- 172.21.15.2:0/1624700986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 msgr2=0x7f931010eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:37.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 --2- 172.21.15.2:0/1624700986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f931010eb30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f930800b0d0 tx=0x7f930800b3e0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 -- 172.21.15.2:0/1624700986 shutdown_connections 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 --2- 172.21.15.2:0/1624700986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f931010eb30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 --2- 172.21.15.2:0/1624700986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 --2- 172.21.15.2:0/1624700986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f931006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.726+0000 7f9317000700 1 -- 172.21.15.2:0/1624700986 >> 172.21.15.2:0/1624700986 conn(0x7f9310076b90 msgr2=0x7f9310076fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.727+0000 7f9317000700 1 -- 172.21.15.2:0/1624700986 shutdown_connections 2024-03-11T18:29:37.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.727+0000 7f9317000700 1 -- 172.21.15.2:0/1624700986 wait complete. 2024-03-11T18:29:37.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 Processor -- start 2024-03-11T18:29:37.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 -- start start 2024-03-11T18:29:37.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f9310100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f93101011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93101a6740 con 0x7f931006ac90 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93101a68b0 con 0x7f9310107cf0 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.728+0000 7f9317000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93101a6a70 con 0x7f931006a2f0 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f930f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f93101011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f930fe00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f9310100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f930f400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f93101011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33172/0 (socket says 172.21.15.2:33172) 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f930fe00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f9310100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:55162/0 (socket says 172.21.15.2:55162) 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f930f400700 1 -- 172.21.15.2:0/3142812768 learned_addr learned my addr 172.21.15.2:0/3142812768 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 -- 172.21.15.2:0/3142812768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 msgr2=0x7f9310100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:37.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f9310100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 -- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 msgr2=0x7f93101011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f93101011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 -- 172.21.15.2:0/3142812768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9300009520 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.729+0000 7f9315000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101016f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f930800d010 tx=0x7f930800bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.730+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930800e040 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.730+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93080079d0 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.730+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9308009d00 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.730+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930801d480 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.730+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93101a6f20 con 0x7f9310107cf0 2024-03-11T18:29:37.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.731+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931010a210 con 0x7f9310107cf0 2024-03-11T18:29:37.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.733+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f9308003a00 con 0x7f9310107cf0 2024-03-11T18:29:37.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.734+0000 7f930cc00700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 0x7f92f40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:37.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.735+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f93080cdcd0 con 0x7f9310107cf0 2024-03-11T18:29:37.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.735+0000 7f930fe00700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 0x7f92f40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:37.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.735+0000 7f930fe00700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 0x7f92f40a3f80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9300009d70 tx=0x7f9300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.739+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9308097370 con 0x7f9310107cf0 2024-03-11T18:29:37.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.966+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9310111d10 con 0x7f9310107cf0 2024-03-11T18:29:37.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.967+0000 7f930cc00700 1 -- 172.21.15.2:0/3142812768 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f930809aa30 con 0x7f9310107cf0 2024-03-11T18:29:37.969 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 msgr2=0x7f92f40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 0x7f92f40a3f80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9300009d70 tx=0x7f9300009260 comp rx=0 tx=0).stop 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 msgr2=0x7f93101016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101016f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f930800d010 tx=0x7f930800bd50 comp rx=0 tx=0).stop 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 shutdown_connections 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f92f40a1ac0 0x7f92f40a3f80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f931006a2f0 0x7f9310100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9310107cf0 0x7f93101016f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 --2- 172.21.15.2:0/3142812768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f931006ac90 0x7f93101011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 >> 172.21.15.2:0/3142812768 conn(0x7f9310076b90 msgr2=0x7f93100ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 shutdown_connections 2024-03-11T18:29:37.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:37.970+0000 7f9317000700 1 -- 172.21.15.2:0/3142812768 wait complete. 2024-03-11T18:29:38.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:37 smithi165 ceph-mon[36634]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:37 smithi023 ceph-mon[36406]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:37 smithi002 ceph-mon[31328]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:38.732 INFO:tasks.cephadm:Adding mds.a on smithi002 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.d on smithi002 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.g on smithi002 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.j on smithi002 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.b on smithi023 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.e on smithi023 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.h on smithi023 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.k on smithi023 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.c on smithi165 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.f on smithi165 2024-03-11T18:29:38.733 INFO:tasks.cephadm:Adding mds.i on smithi165 2024-03-11T18:29:38.734 INFO:tasks.cephadm:Adding mds.l on smithi165 2024-03-11T18:29:38.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph orch apply mds all '12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l' 2024-03-11T18:29:38.982 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:29:39.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:38 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3142812768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:39.103 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:38 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3142812768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:39.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:38 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3142812768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:40.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:39 smithi023 ceph-mon[36406]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:40.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:39 smithi165 ceph-mon[36634]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:40.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:39 smithi002 ceph-mon[31328]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 -- 172.21.15.165:0/3558757520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 msgr2=0x7f8320104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 --2- 172.21.15.165:0/3558757520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320104b10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f831c009990 tx=0x7f831c009ca0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 -- 172.21.15.165:0/3558757520 shutdown_connections 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 --2- 172.21.15.165:0/3558757520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320104b10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 --2- 172.21.15.165:0/3558757520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320107980 0x7f8320109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 --2- 172.21.15.165:0/3558757520 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8320105050 0x7f8320107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 -- 172.21.15.165:0/3558757520 >> 172.21.15.165:0/3558757520 conn(0x7f83200fc130 msgr2=0x7f83200fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 -- 172.21.15.165:0/3558757520 shutdown_connections 2024-03-11T18:29:40.617 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.614+0000 7f83285ae700 1 -- 172.21.15.165:0/3558757520 wait complete. 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 Processor -- start 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 -- start start 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 0x7f83201981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8320107980 0x7f832019f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832019f7e0 con 0x7f8320107980 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832019f920 con 0x7f8320105050 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f83285ae700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832019fae0 con 0x7f8320102720 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:39974/0 (socket says 172.21.15.165:39974) 2024-03-11T18:29:40.618 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 -- 172.21.15.165:0/3202276330 learned_addr learned my addr 172.21.15.165:0/3202276330 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f8325b49700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 0x7f83201981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 -- 172.21.15.165:0/3202276330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 msgr2=0x7f83201981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 0x7f83201981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 -- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8320107980 msgr2=0x7f832019f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8320107980 0x7f832019f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 -- 172.21.15.165:0/3202276330 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c0095f0 con 0x7f8320102720 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f8325b49700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 0x7f83201981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.615+0000 7f832634a700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f831c00b5c0 tx=0x7f831c00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.616+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c01d070 con 0x7f8320102720 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.616+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832019fd60 con 0x7f8320102720 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.616+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f831c00fc30 con 0x7f8320102720 2024-03-11T18:29:40.619 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.616+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c017650 con 0x7f8320102720 2024-03-11T18:29:40.620 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.616+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83201a0180 con 0x7f8320102720 2024-03-11T18:29:40.620 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.617+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83200fdcb0 con 0x7f8320102720 2024-03-11T18:29:40.621 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.618+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f831c0177b0 con 0x7f8320102720 2024-03-11T18:29:40.621 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.618+0000 7f83177fe700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 0x7f830c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:40.621 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.619+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f831c0ceb80 con 0x7f8320102720 2024-03-11T18:29:40.621 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.619+0000 7f8325b49700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 0x7f830c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:40.621 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.620+0000 7f8325b49700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 0x7f830c0a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8320199170 tx=0x7f8310008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:40.622 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.622+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f831c0982c0 con 0x7f8320102720 2024-03-11T18:29:40.825 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.824+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f832002d970 con 0x7f830c0a1ac0 2024-03-11T18:29:40.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.841+0000 7f83177fe700 1 -- 172.21.15.165:0/3202276330 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f832002d970 con 0x7f830c0a1ac0 2024-03-11T18:29:40.843 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mds.all update... 2024-03-11T18:29:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 msgr2=0x7f830c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 0x7f830c0a3f80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8320199170 tx=0x7f8310008040 comp rx=0 tx=0).stop 2024-03-11T18:29:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 msgr2=0x7f8320197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f831c00b5c0 tx=0x7f831c00f740 comp rx=0 tx=0).stop 2024-03-11T18:29:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 shutdown_connections 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f830c0a1ac0 0x7f830c0a3f80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8320102720 0x7f8320197c60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8320107980 0x7f832019f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 --2- 172.21.15.165:0/3202276330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8320105050 0x7f83201981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 >> 172.21.15.165:0/3202276330 conn(0x7f83200fc130 msgr2=0x7f83201031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 shutdown_connections 2024-03-11T18:29:40.846 INFO:teuthology.orchestra.run.smithi165.stderr:2024-03-11T18:29:40.843+0000 7f83285ae700 1 -- 172.21.15.165:0/3202276330 wait complete. 2024-03-11T18:29:41.496 DEBUG:teuthology.orchestra.run.smithi002:mds.a> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.a.service 2024-03-11T18:29:41.508 DEBUG:teuthology.orchestra.run.smithi002:mds.d> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.d.service 2024-03-11T18:29:41.560 DEBUG:teuthology.orchestra.run.smithi002:mds.g> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.g.service 2024-03-11T18:29:41.605 DEBUG:teuthology.orchestra.run.smithi002:mds.j> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.j.service 2024-03-11T18:29:41.650 DEBUG:teuthology.orchestra.run.smithi023:mds.b> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.b.service 2024-03-11T18:29:41.652 DEBUG:teuthology.orchestra.run.smithi023:mds.e> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.e.service 2024-03-11T18:29:41.696 DEBUG:teuthology.orchestra.run.smithi023:mds.h> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.h.service 2024-03-11T18:29:41.739 DEBUG:teuthology.orchestra.run.smithi023:mds.k> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.k.service 2024-03-11T18:29:41.782 DEBUG:teuthology.orchestra.run.smithi165:mds.c> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.c.service 2024-03-11T18:29:41.784 DEBUG:teuthology.orchestra.run.smithi165:mds.f> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.f.service 2024-03-11T18:29:41.827 DEBUG:teuthology.orchestra.run.smithi165:mds.i> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.i.service 2024-03-11T18:29:41.871 DEBUG:teuthology.orchestra.run.smithi165:mds.l> sudo journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.l.service 2024-03-11T18:29:41.913 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-11T18:29:41.914 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:29:42.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 -- 172.21.15.2:0/1004271489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c102400 msgr2=0x7f291c102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 --2- 172.21.15.2:0/1004271489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c102400 0x7f291c102880 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2908009900 tx=0x7f290802bc30 comp rx=0 tx=0).stop 2024-03-11T18:29:42.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 -- 172.21.15.2:0/1004271489 shutdown_connections 2024-03-11T18:29:42.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 --2- 172.21.15.2:0/1004271489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102dc0 0x7f291c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 --2- 172.21.15.2:0/1004271489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c102400 0x7f291c102880 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 --2- 172.21.15.2:0/1004271489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c101210 0x7f291c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.153+0000 7f2923200640 1 -- 172.21.15.2:0/1004271489 >> 172.21.15.2:0/1004271489 conn(0x7f291c0fc7a0 msgr2=0x7f291c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:42.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.154+0000 7f2923200640 1 -- 172.21.15.2:0/1004271489 shutdown_connections 2024-03-11T18:29:42.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.154+0000 7f2923200640 1 -- 172.21.15.2:0/1004271489 wait complete. 2024-03-11T18:29:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:41 smithi023 ceph-mon[36406]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:41 smithi023 ceph-mon[36406]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:41 smithi023 ceph-mon[36406]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l;count:12 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:41 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:41 smithi165 ceph-mon[36634]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:41 smithi165 ceph-mon[36634]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:41 smithi165 ceph-mon[36634]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l;count:12 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:42.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:41 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:42.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.155+0000 7f2923200640 1 Processor -- start 2024-03-11T18:29:42.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.155+0000 7f2923200640 1 -- start start 2024-03-11T18:29:42.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.155+0000 7f2923200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.155+0000 7f2923200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102400 0x7f291c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.155+0000 7f2923200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c102dc0 0x7f291c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f2923200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a0f20 con 0x7f291c102dc0 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f2923200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a1090 con 0x7f291c101210 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f2923200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1a1250 con 0x7f291c102400 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45410/0 (socket says 172.21.15.2:45410) 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 -- 172.21.15.2:0/3614324370 learned_addr learned my addr 172.21.15.2:0/3614324370 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291b400640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102400 0x7f291c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f2921200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c102dc0 0x7f291c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 -- 172.21.15.2:0/3614324370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102400 msgr2=0x7f291c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102400 0x7f291c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 -- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c102dc0 msgr2=0x7f291c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c102dc0 0x7f291c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.156+0000 7f291be00640 1 -- 172.21.15.2:0/3614324370 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c009490 con 0x7f291c101210 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f291be00640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f290c00e9c0 tx=0x7f290c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c00cde0 con 0x7f291c101210 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f290c010910 con 0x7f291c101210 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c018a10 con 0x7f291c101210 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2914009840 con 0x7f291c101210 2024-03-11T18:29:42.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.157+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2908009560 con 0x7f291c101210 2024-03-11T18:29:42.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.158+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c10cc00 con 0x7f291c101210 2024-03-11T18:29:42.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.160+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f290c010a80 con 0x7f291c101210 2024-03-11T18:29:42.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.161+0000 7f2918c00640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 0x7f29000a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.161+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f290c014070 con 0x7f291c101210 2024-03-11T18:29:42.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.161+0000 7f291b400640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 0x7f29000a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.162+0000 7f291b400640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 0x7f29000a3f70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2908000c00 tx=0x7f290802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:42.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.165+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f290c096cf0 con 0x7f291c101210 2024-03-11T18:29:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:41 smithi002 ceph-mon[31328]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:41 smithi002 ceph-mon[31328]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:29:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:41 smithi002 ceph-mon[31328]: Saving service mds.all spec with placement smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l;count:12 2024-03-11T18:29:42.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:42.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:41 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:29:42.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.368+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f291c1a1b10 con 0x7f291c101210 2024-03-11T18:29:42.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.370+0000 7f2918c00640 1 -- 172.21.15.2:0/3614324370 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18375 (secure 0 0 0) 0x7f290c09a3b0 con 0x7f291c101210 2024-03-11T18:29:42.372 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:42.372 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:29:30.822013+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 msgr2=0x7f29000a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 0x7f29000a3f70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2908000c00 tx=0x7f290802c7e0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 msgr2=0x7f291c199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f290c00e9c0 tx=0x7f290c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 shutdown_connections 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29000a1ab0 0x7f29000a3f70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c102dc0 0x7f291c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f291c102400 0x7f291c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 --2- 172.21.15.2:0/3614324370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f291c101210 0x7f291c199430 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 >> 172.21.15.2:0/3614324370 conn(0x7f291c0fc7a0 msgr2=0x7f291c0733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 shutdown_connections 2024-03-11T18:29:42.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.373+0000 7f2923200640 1 -- 172.21.15.2:0/3614324370 wait complete. 2024-03-11T18:29:42.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T18:29:42.395 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:29:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 -- 172.21.15.2:0/1657904921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c073be0 msgr2=0x7f5b4c074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 --2- 172.21.15.2:0/1657904921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c073be0 0x7f5b4c074060 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009ca0 tx=0x7f5b3402c100 comp rx=0 tx=0).stop 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 -- 172.21.15.2:0/1657904921 shutdown_connections 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 --2- 172.21.15.2:0/1657904921 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c10f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 --2- 172.21.15.2:0/1657904921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c073be0 0x7f5b4c074060 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 --2- 172.21.15.2:0/1657904921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 0x7f5b4c073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 -- 172.21.15.2:0/1657904921 >> 172.21.15.2:0/1657904921 conn(0x7f5b4c0fc540 msgr2=0x7f5b4c0fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 -- 172.21.15.2:0/1657904921 shutdown_connections 2024-03-11T18:29:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.617+0000 7f5b50a00640 1 -- 172.21.15.2:0/1657904921 wait complete. 2024-03-11T18:29:42.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.618+0000 7f5b50a00640 1 Processor -- start 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 -- start start 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 0x7f5b4c19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c1a5130 con 0x7f5b4c073be0 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c1a52a0 con 0x7f5b4c0751c0 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b50a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c1a5460 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b4b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b4be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b4b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33224/0 (socket says 172.21.15.2:33224) 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b4b400640 1 -- 172.21.15.2:0/2578028805 learned_addr learned my addr 172.21.15.2:0/2578028805 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.619+0000 7f5b4aa00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 0x7f5b4c19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 -- 172.21.15.2:0/2578028805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 msgr2=0x7f5b4c19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 0x7f5b4c19db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 -- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 msgr2=0x7f5b4c19d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 -- 172.21.15.2:0/2578028805 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b34009900 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4b400640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b4be00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c1a4bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400d680 tx=0x7f5b4400db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44005220 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b44009d00 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44004ea0 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.620+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c1a5740 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.621+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c1a5c20 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.622+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c103660 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.624+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f5b44009820 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.626+0000 7f5b43e00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 0x7f5b2c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:42.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.626+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5b440cc810 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.626+0000 7f5b4b400640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 0x7f5b2c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:42.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.627+0000 7f5b4b400640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 0x7f5b2c0a3f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c011500 tx=0x7f5b3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:42.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.633+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5b44095ed0 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.834+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b4c02ce40 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.834+0000 7f5b43e00640 1 -- 172.21.15.2:0/2578028805 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b44099590 con 0x7f5b4c10d5a0 2024-03-11T18:29:42.836 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 msgr2=0x7f5b2c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 0x7f5b2c0a3f70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c011500 tx=0x7f5b3c005f90 comp rx=0 tx=0).stop 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 msgr2=0x7f5b4c1a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c1a4bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400d680 tx=0x7f5b4400db40 comp rx=0 tx=0).stop 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 shutdown_connections 2024-03-11T18:29:42.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5b2c0a1ab0 0x7f5b2c0a3f70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b4c10d5a0 0x7f5b4c1a4bf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b4c0751c0 0x7f5b4c19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 --2- 172.21.15.2:0/2578028805 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b4c073be0 0x7f5b4c19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.837+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 >> 172.21.15.2:0/2578028805 conn(0x7f5b4c0fc540 msgr2=0x7f5b4c100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.838+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 shutdown_connections 2024-03-11T18:29:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:42.838+0000 7f5b50a00640 1 -- 172.21.15.2:0/2578028805 wait complete. 2024-03-11T18:29:42.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:43.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 -- 172.21.15.2:0/3656356857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 msgr2=0x7f3c04077fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:43.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 --2- 172.21.15.2:0/3656356857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c04077fc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009ca0 tx=0x7f3bfc02c100 comp rx=0 tx=0).stop 2024-03-11T18:29:43.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 -- 172.21.15.2:0/3656356857 shutdown_connections 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 --2- 172.21.15.2:0/3656356857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c04077fc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 --2- 172.21.15.2:0/3656356857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c040771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 --2- 172.21.15.2:0/3656356857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c04075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 -- 172.21.15.2:0/3656356857 >> 172.21.15.2:0/3656356857 conn(0x7f3c040fe990 msgr2=0x7f3c04100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 -- 172.21.15.2:0/3656356857 shutdown_connections 2024-03-11T18:29:43.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.083+0000 7f3c09200640 1 -- 172.21.15.2:0/3656356857 wait complete. 2024-03-11T18:29:43.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.084+0000 7f3c09200640 1 Processor -- start 2024-03-11T18:29:43.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.084+0000 7f3c09200640 1 -- start start 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c0419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c0419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c041a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a52b0 con 0x7f3c04075b40 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a5420 con 0x7f3c04077720 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c09200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a55e0 con 0x7f3c04076d80 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c0419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c0419d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33234/0 (socket says 172.21.15.2:33234) 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01e00640 1 -- 172.21.15.2:0/1416102686 learned_addr learned my addr 172.21.15.2:0/1416102686 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c0419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c02800640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c041a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 -- 172.21.15.2:0/1416102686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 msgr2=0x7f3c041a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:43.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c041a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 -- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 msgr2=0x7f3c0419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c0419d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.085+0000 7f3c01400640 1 -- 172.21.15.2:0/1416102686 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0009490 con 0x7f3c04076d80 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3c01400640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c0419dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000e9f0 tx=0x7f3bf000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000cd80 con 0x7f3c04076d80 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf00035f0 con 0x7f3c04076d80 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0010690 con 0x7f3c04076d80 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc009900 con 0x7f3c04076d80 2024-03-11T18:29:43.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.086+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c041a5c90 con 0x7f3c04076d80 2024-03-11T18:29:43.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.087+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0410fef0 con 0x7f3c04076d80 2024-03-11T18:29:43.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.091+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3bf0003af0 con 0x7f3c04076d80 2024-03-11T18:29:43.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.092+0000 7f3bfaa00640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 0x7f3be80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:43.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.092+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3bf0014070 con 0x7f3c04076d80 2024-03-11T18:29:43.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.093+0000 7f3c01e00640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 0x7f3be80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:43.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.093+0000 7f3c01e00640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 0x7f3be80a3f70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4005fd0 tx=0x7f3bf400b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:43.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.096+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bf0096970 con 0x7f3c04076d80 2024-03-11T18:29:43.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.101 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3614324370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:43.102 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:42 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2578028805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3614324370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:43.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:42 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2578028805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3614324370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:29:43.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:42 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2578028805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:29:43.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.344+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c041131d0 con 0x7f3c04076d80 2024-03-11T18:29:43.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.345+0000 7f3bfaa00640 1 -- 172.21.15.2:0/1416102686 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f3bf009a030 con 0x7f3c04076d80 2024-03-11T18:29:43.346 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:43.346 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-11T18:29:43.347 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 1 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 msgr2=0x7f3be80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 0x7f3be80a3f70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4005fd0 tx=0x7f3bf400b480 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 msgr2=0x7f3c0419dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c0419dd00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000e9f0 tx=0x7f3bf000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 shutdown_connections 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1ab0 0x7f3be80a3f70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04077720 0x7f3c041a4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04076d80 0x7f3c0419dd00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 --2- 172.21.15.2:0/1416102686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04075b40 0x7f3c0419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 >> 172.21.15.2:0/1416102686 conn(0x7f3c040fe990 msgr2=0x7f3c040fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:43.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.348+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 shutdown_connections 2024-03-11T18:29:43.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:43.349+0000 7f3c09200640 1 -- 172.21.15.2:0/1416102686 wait complete. 2024-03-11T18:29:44.086 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:43 smithi165 ceph-mon[36634]: Deploying daemon mds.l on smithi165 2024-03-11T18:29:44.086 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:43 smithi165 ceph-mon[36634]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:44.087 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:43 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1416102686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:43 smithi023 ceph-mon[36406]: Deploying daemon mds.l on smithi165 2024-03-11T18:29:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:43 smithi023 ceph-mon[36406]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:43 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1416102686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:44.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:43 smithi002 ceph-mon[31328]: Deploying daemon mds.l on smithi165 2024-03-11T18:29:44.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:43 smithi002 ceph-mon[31328]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:44.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:43 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1416102686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:44.405 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 systemd[1]: Starting Ceph mds.l for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 podman[66734]: 2024-03-11 18:29:44.383909246 +0000 UTC m=+0.017700154 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 podman[66734]: 2024-03-11 18:29:44.490725676 +0000 UTC m=+0.124516613 container create 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 podman[66734]: 2024-03-11 18:29:44.693166902 +0000 UTC m=+0.326957812 container init 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 podman[66734]: 2024-03-11 18:29:44.695943414 +0000 UTC m=+0.329734313 container start 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 ceph-mds[66753]: -- 172.21.15.165:0/3431691905 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b7451f41a0 con 0x55b7451cb000 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l[66749]: starting mds.l at 2024-03-11T18:29:44.778 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 bash[66734]: 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b 2024-03-11T18:29:45.155 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:29:44 smithi165 systemd[1]: Started Ceph mds.l for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:29:45.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:45.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 -- 172.21.15.2:0/3604801381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a41032a0 msgr2=0x7fc7a41036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:45.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 --2- 172.21.15.2:0/3604801381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a41032a0 0x7fc7a41036c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc794009900 tx=0x7fc79402bc40 comp rx=0 tx=0).stop 2024-03-11T18:29:45.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 -- 172.21.15.2:0/3604801381 shutdown_connections 2024-03-11T18:29:45.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 --2- 172.21.15.2:0/3604801381 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104e50 0x7fc7a410b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 --2- 172.21.15.2:0/3604801381 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a4104490 0x7fc7a4104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 --2- 172.21.15.2:0/3604801381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a41032a0 0x7fc7a41036c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.605+0000 7fc7a3200640 1 -- 172.21.15.2:0/3604801381 >> 172.21.15.2:0/3604801381 conn(0x7fc7a40fe830 msgr2=0x7fc7a4100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:45.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.606+0000 7fc7a3200640 1 -- 172.21.15.2:0/3604801381 shutdown_connections 2024-03-11T18:29:45.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.606+0000 7fc7a3200640 1 -- 172.21.15.2:0/3604801381 wait complete. 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 Processor -- start 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 -- start start 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 0x7fc7a4199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 0x7fc7a4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a4104e50 0x7fc7a41a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:45.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41a0df0 con 0x7fc7a41032a0 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41a0f60 con 0x7fc7a4104490 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a3200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a41a1120 con 0x7fc7a4104e50 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 0x7fc7a4199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc79be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 0x7fc7a4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 0x7fc7a4199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33248/0 (socket says 172.21.15.2:33248) 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a1e00640 1 -- 172.21.15.2:0/1873339890 learned_addr learned my addr 172.21.15.2:0/1873339890 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.607+0000 7fc7a2800640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a4104e50 0x7fc7a41a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 -- 172.21.15.2:0/1873339890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a4104e50 msgr2=0x7fc7a41a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a4104e50 0x7fc7a41a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 -- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 msgr2=0x7fc7a4199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 0x7fc7a4199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 -- 172.21.15.2:0/1873339890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794009560 con 0x7fc7a4104490 2024-03-11T18:29:45.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79be00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 0x7fc7a4199840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c009c30 con 0x7fc7a4104490 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc78c004510 con 0x7fc7a4104490 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a41a1400 con 0x7fc7a4104490 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.608+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c003e60 con 0x7fc7a4104490 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.609+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a41a1960 con 0x7fc7a4104490 2024-03-11T18:29:45.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.610+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a410a560 con 0x7fc7a4104490 2024-03-11T18:29:45.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.611+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc78c003fc0 con 0x7fc7a4104490 2024-03-11T18:29:45.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.613+0000 7fc79aa00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 0x7fc7880a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:45.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.613+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc78c0cc9b0 con 0x7fc7a4104490 2024-03-11T18:29:45.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.613+0000 7fc7a1e00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 0x7fc7880a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:45.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.614+0000 7fc7a1e00640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 0x7fc7880a3f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc794006010 tx=0x7fc79402c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:45.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.615+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc78c096190 con 0x7fc7a4104490 2024-03-11T18:29:45.818 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6832/4285430616,v1:172.21.15.165:6833/4285430616] up:boot 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: fsmap 1 up:standby 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:45.819 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:45 smithi023 ceph-mon[36406]: Deploying daemon mds.k on smithi023 2024-03-11T18:29:45.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.858+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a410ed00 con 0x7fc7a4104490 2024-03-11T18:29:45.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.859+0000 7fc79aa00640 1 -- 172.21.15.2:0/1873339890 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fc78c020090 con 0x7fc7a4104490 2024-03-11T18:29:45.860 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:45.861 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T18:29:45.861 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2024-03-11T18:29:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 msgr2=0x7fc7880a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 0x7fc7880a3f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc794006010 tx=0x7fc79402c7f0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 msgr2=0x7fc7a4199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 0x7fc7a4199840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c00dde0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 shutdown_connections 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7880a1aa0 0x7fc7880a3f60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7a4104e50 0x7fc7a41a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7a4104490 0x7fc7a4199840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 --2- 172.21.15.2:0/1873339890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7a41032a0 0x7fc7a4199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 >> 172.21.15.2:0/1873339890 conn(0x7fc7a40fe830 msgr2=0x7fc7a4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 shutdown_connections 2024-03-11T18:29:45.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:45.862+0000 7fc7a3200640 1 -- 172.21.15.2:0/1873339890 wait complete. 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6832/4285430616,v1:172.21.15.165:6833/4285430616] up:boot 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: fsmap 1 up:standby 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:46.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:46.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:45 smithi002 ceph-mon[31328]: Deploying daemon mds.k on smithi023 2024-03-11T18:29:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6832/4285430616,v1:172.21.15.165:6833/4285430616] up:boot 2024-03-11T18:29:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: fsmap 1 up:standby 2024-03-11T18:29:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:29:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:45 smithi165 ceph-mon[36634]: Deploying daemon mds.k on smithi023 2024-03-11T18:29:46.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1873339890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:47.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1873339890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:47.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1873339890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:47.655 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:47 smithi023 podman[66804]: 2024-03-11 18:29:47.489052055 +0000 UTC m=+0.018502651 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:29:47.883 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:48.006 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:47 smithi023 ceph-mon[36406]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:48.006 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:47 smithi023 podman[66804]: 2024-03-11 18:29:47.733582389 +0000 UTC m=+0.263032963 container create dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T18:29:48.007 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:47 smithi023 podman[66804]: 2024-03-11 18:29:47.978174373 +0000 UTC m=+0.507624953 container init dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True) 2024-03-11T18:29:48.007 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:47 smithi023 podman[66804]: 2024-03-11 18:29:47.981051623 +0000 UTC m=+0.510502204 container start dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 -- 172.21.15.2:0/1332674841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254106d70 msgr2=0x7ff2541091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 --2- 172.21.15.2:0/1332674841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254106d70 0x7ff2541091c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009be0 tx=0x7ff24c02bef0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 -- 172.21.15.2:0/1332674841 shutdown_connections 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 --2- 172.21.15.2:0/1332674841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254106d70 0x7ff2541091c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 --2- 172.21.15.2:0/1332674841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254104440 0x7ff254106830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 --2- 172.21.15.2:0/1332674841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254101b10 0x7ff254103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 -- 172.21.15.2:0/1332674841 >> 172.21.15.2:0/1332674841 conn(0x7ff2540fb500 msgr2=0x7ff2540fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 -- 172.21.15.2:0/1332674841 shutdown_connections 2024-03-11T18:29:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.118+0000 7ff258e00640 1 -- 172.21.15.2:0/1332674841 wait complete. 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.119+0000 7ff258e00640 1 Processor -- start 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.119+0000 7ff258e00640 1 -- start start 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254106d70 0x7ff2541a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2541a0e00 con 0x7ff254104440 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2541a0f70 con 0x7ff254101b10 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff258e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2541a1130 con 0x7ff254106d70 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff252a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff253400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff253400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54616/0 (socket says 172.21.15.2:54616) 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff252a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60438/0 (socket says 172.21.15.2:60438) 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff253400640 1 -- 172.21.15.2:0/699612429 learned_addr learned my addr 172.21.15.2:0/699612429 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.120+0000 7ff253e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254106d70 0x7ff2541a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 -- 172.21.15.2:0/699612429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254106d70 msgr2=0x7ff2541a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254106d70 0x7ff2541a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 -- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 msgr2=0x7ff254199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 -- 172.21.15.2:0/699612429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c009840 con 0x7ff254101b10 2024-03-11T18:29:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff252a00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.121+0000 7ff253400640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e9c0 tx=0x7ff24400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.123+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400ccc0 con 0x7ff254101b10 2024-03-11T18:29:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.123+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff24400ce20 con 0x7ff254101b10 2024-03-11T18:29:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.123+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244010430 con 0x7ff254101b10 2024-03-11T18:29:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.123+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2541a1410 con 0x7ff254101b10 2024-03-11T18:29:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.123+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2541a1920 con 0x7ff254101b10 2024-03-11T18:29:48.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.125+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff254108860 con 0x7ff254101b10 2024-03-11T18:29:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.127+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7ff244005020 con 0x7ff254101b10 2024-03-11T18:29:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.128+0000 7ff24be00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 0x7ff2380a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:48.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.128+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff244014070 con 0x7ff254101b10 2024-03-11T18:29:48.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.128+0000 7ff252a00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 0x7ff2380a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:48.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.129+0000 7ff252a00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 0x7ff2380a4030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff25419a820 tx=0x7ff240005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:48.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.133+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff2440967c0 con 0x7ff254101b10 2024-03-11T18:29:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:47 smithi165 ceph-mon[36634]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:48.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:47 smithi002 ceph-mon[31328]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.398+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25410caa0 con 0x7ff254101b10 2024-03-11T18:29:48.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.399+0000 7ff24be00640 1 -- 172.21.15.2:0/699612429 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7ff25410caa0 con 0x7ff254101b10 2024-03-11T18:29:48.400 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:48.401 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-11T18:29:48.401 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2024-03-11T18:29:48.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 msgr2=0x7ff2380a4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 0x7ff2380a4030 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff25419a820 tx=0x7ff240005f90 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 msgr2=0x7ff254199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff24400e9c0 tx=0x7ff24400ee80 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 shutdown_connections 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff2380a1b70 0x7ff2380a4030 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff254106d70 0x7ff2541a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff254104440 0x7ff254199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 --2- 172.21.15.2:0/699612429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff254101b10 0x7ff254199310 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 >> 172.21.15.2:0/699612429 conn(0x7ff2540fb500 msgr2=0x7ff2541011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 shutdown_connections 2024-03-11T18:29:48.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:48.402+0000 7ff258e00640 1 -- 172.21.15.2:0/699612429 wait complete. 2024-03-11T18:29:48.405 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:48 smithi023 ceph-mds[66823]: -- 172.21.15.23:0/3677981667 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559cfd2ca1a0 con 0x559cfd2a0800 2024-03-11T18:29:48.405 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:48 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k[66819]: starting mds.k at 2024-03-11T18:29:48.405 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:48 smithi023 bash[66804]: dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da 2024-03-11T18:29:48.405 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:29:48 smithi023 systemd[1]: Started Ceph mds.k for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:49.395 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: Deploying daemon mds.j on smithi002 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/699612429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6832/2120271757,v1:172.21.15.23:6833/2120271757] up:boot 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: fsmap 2 up:standby 2024-03-11T18:29:49.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:49 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: Deploying daemon mds.j on smithi002 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/699612429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:49.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6832/2120271757,v1:172.21.15.23:6833/2120271757] up:boot 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: fsmap 2 up:standby 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:49 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:49.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:49.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: Deploying daemon mds.j on smithi002 2024-03-11T18:29:49.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/699612429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:49.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6832/2120271757,v1:172.21.15.23:6833/2120271757] up:boot 2024-03-11T18:29:49.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: fsmap 2 up:standby 2024-03-11T18:29:49.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:49 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:29:50.433 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 -- 172.21.15.2:0/3577324341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c108890 msgr2=0x7f4f3c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 --2- 172.21.15.2:0/3577324341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c108890 0x7f4f3c108d10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009900 tx=0x7f4f2802bc30 comp rx=0 tx=0).stop 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 -- 172.21.15.2:0/3577324341 shutdown_connections 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 --2- 172.21.15.2:0/3577324341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 0x7f4f3c10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 --2- 172.21.15.2:0/3577324341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c108890 0x7f4f3c108d10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 --2- 172.21.15.2:0/3577324341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 -- 172.21.15.2:0/3577324341 >> 172.21.15.2:0/3577324341 conn(0x7f4f3c076640 msgr2=0x7f4f3c078a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 -- 172.21.15.2:0/3577324341 shutdown_connections 2024-03-11T18:29:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.667+0000 7f4f43000640 1 -- 172.21.15.2:0/3577324341 wait complete. 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.668+0000 7f4f43000640 1 Processor -- start 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.668+0000 7f4f43000640 1 -- start start 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 0x7f4f3c19dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 0x7f4f3c1a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1a5200 con 0x7f4f3c1076a0 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1a5370 con 0x7f4f3c108890 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f43000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c1a5530 con 0x7f4f3c109250 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:50.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60456/0 (socket says 172.21.15.2:60456) 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41c00640 1 -- 172.21.15.2:0/4269632705 learned_addr learned my addr 172.21.15.2:0/4269632705 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f42600640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 0x7f4f3c1a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 0x7f4f3c19dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 -- 172.21.15.2:0/4269632705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 msgr2=0x7f4f3c1a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 0x7f4f3c1a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 -- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 msgr2=0x7f4f3c19d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41200640 1 -- 172.21.15.2:0/4269632705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009490 con 0x7f4f3c108890 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.669+0000 7f4f41c00640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f41200640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 0x7f4f3c19dc50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f28000c00 tx=0x7f4f28005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f280043d0 con 0x7f4f3c108890 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f280052d0 con 0x7f4f3c108890 2024-03-11T18:29:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28003d40 con 0x7f4f3c108890 2024-03-11T18:29:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f28009560 con 0x7f4f3c108890 2024-03-11T18:29:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.670+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c1a5ab0 con 0x7f4f3c108890 2024-03-11T18:29:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.671+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c10e9c0 con 0x7f4f3c108890 2024-03-11T18:29:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.673+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f4f28005440 con 0x7f4f3c108890 2024-03-11T18:29:50.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.675+0000 7f4f3a000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 0x7f4f240a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:50.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.675+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4f280ef290 con 0x7f4f3c108890 2024-03-11T18:29:50.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.675+0000 7f4f41c00640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 0x7f4f240a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:50.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.676+0000 7f4f41c00640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 0x7f4f240a3f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4f300058c0 tx=0x7f4f30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:50.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.677+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f280b88a0 con 0x7f4f3c108890 2024-03-11T18:29:50.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.940+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3c1131d0 con 0x7f4f3c108890 2024-03-11T18:29:50.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.941+0000 7f4f3a000640 1 -- 172.21.15.2:0/4269632705 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7f4f280b8030 con 0x7f4f3c108890 2024-03-11T18:29:50.944 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:50.944 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T18:29:50.944 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2024-03-11T18:29:50.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.943+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 msgr2=0x7f4f240a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 0x7f4f240a3f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4f300058c0 tx=0x7f4f30005f90 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 msgr2=0x7f4f3c19dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 0x7f4f3c19dc50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f28000c00 tx=0x7f4f28005e50 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 shutdown_connections 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4f240a1aa0 0x7f4f240a3f60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f3c109250 0x7f4f3c1a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f3c108890 0x7f4f3c19dc50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 --2- 172.21.15.2:0/4269632705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f3c1076a0 0x7f4f3c19d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 >> 172.21.15.2:0/4269632705 conn(0x7f4f3c076640 msgr2=0x7f4f3c10bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 shutdown_connections 2024-03-11T18:29:50.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:50.944+0000 7f4f43000640 1 -- 172.21.15.2:0/4269632705 wait complete. 2024-03-11T18:29:51.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:51 smithi002 ceph-mon[31328]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:51.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:51 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4269632705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:51.503 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:51 smithi002 systemd[1]: Starting Ceph mds.j for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:29:51.556 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:51 smithi023 ceph-mon[36406]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:51.557 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:51 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4269632705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:51.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:51 smithi165 ceph-mon[36634]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:51.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:51 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4269632705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:51.976 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:51 smithi002 podman[72540]: 2024-03-11 18:29:51.600269296 +0000 UTC m=+0.035354355 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:29:51.976 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:51 smithi002 podman[72540]: 2024-03-11 18:29:51.718700863 +0000 UTC m=+0.153785940 container create 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:29:51.976 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:51 smithi002 podman[72540]: 2024-03-11 18:29:51.97264801 +0000 UTC m=+0.407733088 container init 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T18:29:52.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:51 smithi002 podman[72540]: 2024-03-11 18:29:51.983644132 +0000 UTC m=+0.418729226 container start 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T18:29:52.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:52 smithi002 ceph-mds[72560]: -- 172.21.15.2:0/3231794079 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564c79621a0 con 0x5564c7939000 2024-03-11T18:29:52.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:52 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j[72556]: starting mds.j at 2024-03-11T18:29:52.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:52 smithi002 bash[72540]: 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc 2024-03-11T18:29:52.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:29:52 smithi002 systemd[1]: Started Ceph mds.j for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:29:52.966 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 -- 172.21.15.2:0/2377592263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590102dc0 msgr2=0x7fb590109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 --2- 172.21.15.2:0/2377592263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590102dc0 0x7fb590109650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb588009900 tx=0x7fb58802bc30 comp rx=0 tx=0).stop 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 -- 172.21.15.2:0/2377592263 shutdown_connections 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 --2- 172.21.15.2:0/2377592263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590102dc0 0x7fb590109650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 --2- 172.21.15.2:0/2377592263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb590102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 --2- 172.21.15.2:0/2377592263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590101210 0x7fb590101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 -- 172.21.15.2:0/2377592263 >> 172.21.15.2:0/2377592263 conn(0x7fb5900fc7a0 msgr2=0x7fb5900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:53.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 -- 172.21.15.2:0/2377592263 shutdown_connections 2024-03-11T18:29:53.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.179+0000 7fb595e00640 1 -- 172.21.15.2:0/2377592263 wait complete. 2024-03-11T18:29:53.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.180+0000 7fb595e00640 1 Processor -- start 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.180+0000 7fb595e00640 1 -- start start 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 0x7fb5901993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb5901998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 0x7fb5901a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a0ea0 con 0x7fb590102dc0 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a1010 con 0x7fb590102400 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb595e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a11d0 con 0x7fb590101210 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 0x7fb5901a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 0x7fb5901993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58e000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb5901998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 0x7fb5901a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60488/0 (socket says 172.21.15.2:60488) 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58f400640 1 -- 172.21.15.2:0/3004117553 learned_addr learned my addr 172.21.15.2:0/3004117553 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:53.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.181+0000 7fb58ea00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 0x7fb5901993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:35346/0 (socket says 172.21.15.2:35346) 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 -- 172.21.15.2:0/3004117553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 msgr2=0x7fb5901993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 0x7fb5901993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 -- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 msgr2=0x7fb5901a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 0x7fb5901a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 -- 172.21.15.2:0/3004117553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588009560 con 0x7fb590102400 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb58e000640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb5901998f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb57800e9f0 tx=0x7fb57800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57800ccc0 con 0x7fb590102400 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb57800ce20 con 0x7fb590102400 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578004a00 con 0x7fb590102400 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5901a14b0 con 0x7fb590102400 2024-03-11T18:29:53.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.182+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5901a1970 con 0x7fb590102400 2024-03-11T18:29:53.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.183+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59010cc00 con 0x7fb590102400 2024-03-11T18:29:53.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.185+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fb5780050b0 con 0x7fb590102400 2024-03-11T18:29:53.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.187+0000 7fb594a00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 0x7fb5740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:53.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.187+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb578014070 con 0x7fb590102400 2024-03-11T18:29:53.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.187+0000 7fb58ea00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 0x7fb5740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:53.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.188+0000 7fb58ea00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 0x7fb5740a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb590102260 tx=0x7fb580009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:53.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.191+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb578096530 con 0x7fb590102400 2024-03-11T18:29:53.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:53.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.311 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.2:6834/2188580308,v1:172.21.15.2:6835/2188580308] up:boot 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: fsmap 3 up:standby 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:53.312 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:53 smithi002 ceph-mon[31328]: Deploying daemon mds.d on smithi002 2024-03-11T18:29:53.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.447+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5901a1bf0 con 0x7fb590102400 2024-03-11T18:29:53.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.448+0000 7fb594a00640 1 -- 172.21.15.2:0/3004117553 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7fb578099bf0 con 0x7fb590102400 2024-03-11T18:29:53.450 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:53.450 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T18:29:53.450 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 msgr2=0x7fb5740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 0x7fb5740a3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb590102260 tx=0x7fb580009720 comp rx=0 tx=0).stop 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 msgr2=0x7fb5901998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb5901998f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb57800e9f0 tx=0x7fb57800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 shutdown_connections 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5740a1a30 0x7fb5740a3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb590102dc0 0x7fb5901a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb590102400 0x7fb5901998f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 --2- 172.21.15.2:0/3004117553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb590101210 0x7fb5901993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:53.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.450+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 >> 172.21.15.2:0/3004117553 conn(0x7fb5900fc7a0 msgr2=0x7fb5900733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:53.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.451+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 shutdown_connections 2024-03-11T18:29:53.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:53.451+0000 7fb595e00640 1 -- 172.21.15.2:0/3004117553 wait complete. 2024-03-11T18:29:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.2:6834/2188580308,v1:172.21.15.2:6835/2188580308] up:boot 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: fsmap 3 up:standby 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:53 smithi165 ceph-mon[36634]: Deploying daemon mds.d on smithi002 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:53.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.2:6834/2188580308,v1:172.21.15.2:6835/2188580308] up:boot 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: fsmap 3 up:standby 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:53.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:53 smithi023 ceph-mon[36406]: Deploying daemon mds.d on smithi002 2024-03-11T18:29:54.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:54 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3004117553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:54.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:54 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3004117553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:54.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:54 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3004117553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:55.341 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:55 smithi002 ceph-mon[31328]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:55.472 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:55 smithi165 ceph-mon[36634]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:55.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:55 smithi023 ceph-mon[36406]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 -- 172.21.15.2:0/3527550331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 msgr2=0x7f2634074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:55.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 --2- 172.21.15.2:0/3527550331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634074060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f261c009900 tx=0x7f261c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 -- 172.21.15.2:0/3527550331 shutdown_connections 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 --2- 172.21.15.2:0/3527550331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f263410b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 --2- 172.21.15.2:0/3527550331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634074060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 --2- 172.21.15.2:0/3527550331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 -- 172.21.15.2:0/3527550331 >> 172.21.15.2:0/3527550331 conn(0x7f26340fc760 msgr2=0x7f26340feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 -- 172.21.15.2:0/3527550331 shutdown_connections 2024-03-11T18:29:55.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.666+0000 7f2639200640 1 -- 172.21.15.2:0/3527550331 wait complete. 2024-03-11T18:29:55.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.667+0000 7f2639200640 1 Processor -- start 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.667+0000 7f2639200640 1 -- start start 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f26341a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26341a0f40 con 0x7f26340751c0 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26341a10b0 con 0x7f2634073be0 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2639200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26341a1270 con 0x7f2634109070 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2631400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2631e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2631e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54702/0 (socket says 172.21.15.2:54702) 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2631e00640 1 -- 172.21.15.2:0/697429572 learned_addr learned my addr 172.21.15.2:0/697429572 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:55.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f26341a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 -- 172.21.15.2:0/697429572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 msgr2=0x7f2634199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 -- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 msgr2=0x7f2634199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2632800640 1 -- 172.21.15.2:0/697429572 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624009840 con 0x7f2634109070 2024-03-11T18:29:55.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.668+0000 7f2631400640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f2632800640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f26341a0a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f262c00e970 tx=0x7f262c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c009620 con 0x7f2634109070 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261c009560 con 0x7f2634109070 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f262c00ccc0 con 0x7f2634109070 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c004a00 con 0x7f2634109070 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.669+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26341a18f0 con 0x7f2634109070 2024-03-11T18:29:55.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.670+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2634107da0 con 0x7f2634109070 2024-03-11T18:29:55.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.672+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f262c005020 con 0x7f2634109070 2024-03-11T18:29:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.674+0000 7f262aa00640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 0x7f26180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.674+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f262c014070 con 0x7f2634109070 2024-03-11T18:29:55.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.674+0000 7f2631e00640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 0x7f26180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:55.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.675+0000 7f2631e00640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 0x7f26180a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2624011630 tx=0x7f2624009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:55.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.678+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f262c096550 con 0x7f2634109070 2024-03-11T18:29:55.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.924+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f263410ee00 con 0x7f2634109070 2024-03-11T18:29:55.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.926+0000 7f262aa00640 1 -- 172.21.15.2:0/697429572 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7f263410ee00 con 0x7f2634109070 2024-03-11T18:29:55.928 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:55.928 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-11T18:29:55.929 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 4 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 msgr2=0x7f26180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 0x7f26180a3ef0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2624011630 tx=0x7f2624009500 comp rx=0 tx=0).stop 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 msgr2=0x7f26341a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f26341a0a00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f262c00e970 tx=0x7f262c00ee30 comp rx=0 tx=0).stop 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 shutdown_connections 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f26180a1a30 0x7f26180a3ef0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2634109070 0x7f26341a0a00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26340751c0 0x7f2634199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 --2- 172.21.15.2:0/697429572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2634073be0 0x7f2634199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:55.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.928+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 >> 172.21.15.2:0/697429572 conn(0x7f26340fc760 msgr2=0x7f2634104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:55.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.929+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 shutdown_connections 2024-03-11T18:29:55.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:55.929+0000 7f2639200640 1 -- 172.21.15.2:0/697429572 wait complete. 2024-03-11T18:29:55.949 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:55 smithi002 podman[72899]: 2024-03-11 18:29:55.71561345 +0000 UTC m=+0.151057919 container create 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:55 smithi002 podman[72899]: 2024-03-11 18:29:55.978055203 +0000 UTC m=+0.413499707 container init 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:55 smithi002 podman[72899]: 2024-03-11 18:29:55.988041158 +0000 UTC m=+0.423485636 container start 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mds[72940]: -- 172.21.15.2:0/577662839 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55acb0c921a0 con 0x55acb0c68c00 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d[72934]: starting mds.d at 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:56 smithi002 bash[72899]: 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd 2024-03-11T18:29:56.249 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:29:56 smithi002 systemd[1]: Started Ceph mds.d for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:29:56.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/697429572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:56.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:56 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:56.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/697429572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:56.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:56 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/697429572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:29:56.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:56 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:29:57.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:57.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:57.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: Deploying daemon mds.e on smithi023 2024-03-11T18:29:57.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.2:6836/2729524869,v1:172.21.15.2:6837/2729524869] up:boot 2024-03-11T18:29:57.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: fsmap 4 up:standby 2024-03-11T18:29:57.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:57 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: Deploying daemon mds.e on smithi023 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.2:6836/2729524869,v1:172.21.15.2:6837/2729524869] up:boot 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: fsmap 4 up:standby 2024-03-11T18:29:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:57 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: Deploying daemon mds.e on smithi023 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.2:6836/2729524869,v1:172.21.15.2:6837/2729524869] up:boot 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: fsmap 4 up:standby 2024-03-11T18:29:57.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:57 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:29:57.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:29:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 -- 172.21.15.2:0/3233649377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c1010a0 msgr2=0x7fe08c1014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 --2- 172.21.15.2:0/3233649377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c1010a0 0x7fe08c1014c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009900 tx=0x7fe07c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:29:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 -- 172.21.15.2:0/3233649377 shutdown_connections 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 --2- 172.21.15.2:0/3233649377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102c50 0x7fe08c1094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 --2- 172.21.15.2:0/3233649377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c102290 0x7fe08c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 --2- 172.21.15.2:0/3233649377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c1010a0 0x7fe08c1014c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.185+0000 7fe08b200640 1 -- 172.21.15.2:0/3233649377 >> 172.21.15.2:0/3233649377 conn(0x7fe08c0fc610 msgr2=0x7fe08c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.186+0000 7fe08b200640 1 -- 172.21.15.2:0/3233649377 shutdown_connections 2024-03-11T18:29:58.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.186+0000 7fe08b200640 1 -- 172.21.15.2:0/3233649377 wait complete. 2024-03-11T18:29:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 Processor -- start 2024-03-11T18:29:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 -- start start 2024-03-11T18:29:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 0x7fe08c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 0x7fe08c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:58.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 0x7fe08c1a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1a0e40 con 0x7fe08c1010a0 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1a0fb0 con 0x7fe08c102c50 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c1a1170 con 0x7fe08c102290 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe08a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 0x7fe08c1a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.187+0000 7fe089e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 0x7fe08c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 0x7fe08c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe08a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 0x7fe08c1a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:33542/0 (socket says 172.21.15.2:33542) 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe08a800640 1 -- 172.21.15.2:0/1852511362 learned_addr learned my addr 172.21.15.2:0/1852511362 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 -- 172.21.15.2:0/1852511362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 msgr2=0x7fe08c1a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 0x7fe08c1a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 -- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 msgr2=0x7fe08c199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 0x7fe08c199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 -- 172.21.15.2:0/1852511362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078009490 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089e00640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 0x7fe08c199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.188+0000 7fe089400640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 0x7fe08c199890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe07800e9f0 tx=0x7fe07800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.189+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07800cd80 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.189+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0780035f0 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.189+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078010690 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.189+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c009560 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.189+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c1a17d0 con 0x7fe08c102290 2024-03-11T18:29:58.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.190+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c10cb00 con 0x7fe08c102290 2024-03-11T18:29:58.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.193+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fe078003af0 con 0x7fe08c102290 2024-03-11T18:29:58.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.194+0000 7fe082a00640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 0x7fe0700a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:29:58.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.194+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe078014070 con 0x7fe08c102290 2024-03-11T18:29:58.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.194+0000 7fe089e00640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 0x7fe0700a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:29:58.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.195+0000 7fe089e00640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 0x7fe0700a3ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c250 tx=0x7fe07c02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:29:58.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.196+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe0780969c0 con 0x7fe08c102290 2024-03-11T18:29:58.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.438+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08c10cc90 con 0x7fe08c102290 2024-03-11T18:29:58.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.439+0000 7fe082a00640 1 -- 172.21.15.2:0/1852511362 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3695 (secure 0 0 0) 0x7fe07809a080 con 0x7fe08c102290 2024-03-11T18:29:58.441 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:29:58.441 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-11T18:29:58.442 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 5 2024-03-11T18:29:58.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 msgr2=0x7fe0700a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:58.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 0x7fe0700a3ee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe07c02c250 tx=0x7fe07c02c7f0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 msgr2=0x7fe08c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 0x7fe08c199890 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe07800e9f0 tx=0x7fe07800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 shutdown_connections 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe0700a1a20 0x7fe0700a3ee0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe08c102c50 0x7fe08c1a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe08c102290 0x7fe08c199890 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 --2- 172.21.15.2:0/1852511362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe08c1010a0 0x7fe08c199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 >> 172.21.15.2:0/1852511362 conn(0x7fe08c0fc610 msgr2=0x7fe08c0733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 shutdown_connections 2024-03-11T18:29:58.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:29:58.442+0000 7fe08b200640 1 -- 172.21.15.2:0/1852511362 wait complete. 2024-03-11T18:29:58.905 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:58 smithi023 systemd[1]: Starting Ceph mds.e for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:29:59.176 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:58 smithi023 podman[67126]: 2024-03-11 18:29:58.813368038 +0000 UTC m=+0.018443019 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:29:59.176 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:58 smithi023 podman[67126]: 2024-03-11 18:29:58.922709701 +0000 UTC m=+0.127784673 container create d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T18:29:59.176 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 podman[67126]: 2024-03-11 18:29:59.175651562 +0000 UTC m=+0.380726544 container init d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:29:59.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:59 smithi002 ceph-mon[31328]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:59.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:29:59 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1852511362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:59.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:59 smithi023 ceph-mon[36406]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:59.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:29:59 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1852511362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:59 smithi165 ceph-mon[36634]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:29:59.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:29:59 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1852511362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:29:59.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 podman[67126]: 2024-03-11 18:29:59.179314273 +0000 UTC m=+0.384389254 container start d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T18:29:59.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 ceph-mds[67144]: -- 172.21.15.23:0/4233280133 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffc01861a0 con 0x55ffc015c800 2024-03-11T18:29:59.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: starting mds.e at 2024-03-11T18:29:59.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 bash[67126]: d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea 2024-03-11T18:29:59.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:29:59 smithi023 systemd[1]: Started Ceph mds.e for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:00.277 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6834/1605486605,v1:172.21.15.23:6835/1605486605] up:boot 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: fsmap 5 up:standby 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: Deploying daemon mds.f on smithi165 2024-03-11T18:30:00.278 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:00 smithi165 ceph-mon[36634]: overall HEALTH_OK 2024-03-11T18:30:00.465 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6834/1605486605,v1:172.21.15.23:6835/1605486605] up:boot 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: fsmap 5 up:standby 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:00.562 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: Deploying daemon mds.f on smithi165 2024-03-11T18:30:00.563 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:00 smithi002 ceph-mon[31328]: overall HEALTH_OK 2024-03-11T18:30:00.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6834/1605486605,v1:172.21.15.23:6835/1605486605] up:boot 2024-03-11T18:30:00.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: fsmap 5 up:standby 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: Deploying daemon mds.f on smithi165 2024-03-11T18:30:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:00 smithi023 ceph-mon[36406]: overall HEALTH_OK 2024-03-11T18:30:00.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1486216038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 msgr2=0x7fc1ac1046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1486216038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac1046e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc198009860 tx=0x7fc19802bb70 comp rx=0 tx=0).stop 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1486216038 shutdown_connections 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1486216038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1486216038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac1046e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1486216038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1486216038 >> 172.21.15.2:0/1486216038 conn(0x7fc1ac0fb500 msgr2=0x7fc1ac0fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1486216038 shutdown_connections 2024-03-11T18:30:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.694+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1486216038 wait complete. 2024-03-11T18:30:00.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.695+0000 7fc1b2c00640 1 Processor -- start 2024-03-11T18:30:00.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.695+0000 7fc1b2c00640 1 -- start start 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.695+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1a5210 con 0x7fc1ac0ff900 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1a5380 con 0x7fc1ac102270 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1a5540 con 0x7fc1ac104c20 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59550/0 (socket says 172.21.15.2:59550) 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 -- 172.21.15.2:0/1478247278 learned_addr learned my addr 172.21.15.2:0/1478247278 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b0e00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2200640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 -- 172.21.15.2:0/1478247278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 msgr2=0x7fc1ac1a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:00.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 -- 172.21.15.2:0/1478247278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 msgr2=0x7fc1ac19dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac19dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b1800640 1 -- 172.21.15.2:0/1478247278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4009900 con 0x7fc1ac0ff900 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b2200640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.696+0000 7fc1b0e00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac19dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1b1800640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9f0 tx=0x7fc1a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0009700 con 0x7fc1ac0ff900 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1a000cd80 con 0x7fc1ac0ff900 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc198009490 con 0x7fc1ac0ff900 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0018920 con 0x7fc1ac0ff900 2024-03-11T18:30:00.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.697+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1a5b70 con 0x7fc1ac0ff900 2024-03-11T18:30:00.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.699+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac0fcff0 con 0x7fc1ac0ff900 2024-03-11T18:30:00.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.700+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc1a0018a80 con 0x7fc1ac0ff900 2024-03-11T18:30:00.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.702+0000 7fc1aa000640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 0x7fc1940a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:00.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.702+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc1a0014070 con 0x7fc1ac0ff900 2024-03-11T18:30:00.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.702+0000 7fc1b0e00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 0x7fc1940a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:00.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.703+0000 7fc1b0e00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 0x7fc1940a3ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc198009830 tx=0x7fc198034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:00.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.706+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1a0052d20 con 0x7fc1ac0ff900 2024-03-11T18:30:00.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.947+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ac02cda0 con 0x7fc1ac0ff900 2024-03-11T18:30:00.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.948+0000 7fc1aa000640 1 -- 172.21.15.2:0/1478247278 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4486 (secure 0 0 0) 0x7fc1a0096a30 con 0x7fc1ac0ff900 2024-03-11T18:30:00.950 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:00.950 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-11T18:30:00.951 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 6 2024-03-11T18:30:00.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.950+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 msgr2=0x7fc1940a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:00.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.950+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 0x7fc1940a3ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc198009830 tx=0x7fc198034be0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.950+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 msgr2=0x7fc1ac19d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.950+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9f0 tx=0x7fc1a000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 shutdown_connections 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc1940a1a20 0x7fc1940a3ee0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1ac104c20 0x7fc1ac1a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1ac102270 0x7fc1ac19dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 --2- 172.21.15.2:0/1478247278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1ac0ff900 0x7fc1ac19d720 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 >> 172.21.15.2:0/1478247278 conn(0x7fc1ac0fb500 msgr2=0x7fc1ac1002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 shutdown_connections 2024-03-11T18:30:00.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:00.951+0000 7fc1b2c00640 1 -- 172.21.15.2:0/1478247278 wait complete. 2024-03-11T18:30:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:01 smithi002 ceph-mon[31328]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:01 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1478247278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:01.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:01 smithi023 ceph-mon[36406]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:01.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:01 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1478247278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:01.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:01 smithi165 ceph-mon[36634]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:01.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:01 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1478247278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:01.656 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:01 smithi165 systemd[1]: Starting Ceph mds.f for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:30:02.097 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:01 smithi165 podman[67044]: 2024-03-11 18:30:01.810745302 +0000 UTC m=+0.103084644 container create b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:30:02.097 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:01 smithi165 podman[67044]: 2024-03-11 18:30:01.725661839 +0000 UTC m=+0.018001171 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:30:02.098 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 podman[67044]: 2024-03-11 18:30:02.013001237 +0000 UTC m=+0.305340569 container init b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T18:30:02.098 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 podman[67044]: 2024-03-11 18:30:02.015614877 +0000 UTC m=+0.307954201 container start b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:30:02.098 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 ceph-mds[67063]: -- 172.21.15.165:0/631312614 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55967d04c1a0 con 0x55967d022800 2024-03-11T18:30:02.098 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: starting mds.f at 2024-03-11T18:30:02.098 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 bash[67044]: b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 2024-03-11T18:30:02.405 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:30:02 smithi165 systemd[1]: Started Ceph mds.f for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:02.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:03.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 -- 172.21.15.2:0/3174214800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741077c0 msgr2=0x7fbe74107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 --2- 172.21.15.2:0/3174214800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741077c0 0x7fbe74107be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbe68009be0 tx=0x7fbe6802bef0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 -- 172.21.15.2:0/3174214800 shutdown_connections 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 --2- 172.21.15.2:0/3174214800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe7410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 --2- 172.21.15.2:0/3174214800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741089b0 0x7fbe74108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 --2- 172.21.15.2:0/3174214800 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741077c0 0x7fbe74107be0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 -- 172.21.15.2:0/3174214800 >> 172.21.15.2:0/3174214800 conn(0x7fbe74076780 msgr2=0x7fbe74078ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 -- 172.21.15.2:0/3174214800 shutdown_connections 2024-03-11T18:30:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.209+0000 7fbe7c600640 1 -- 172.21.15.2:0/3174214800 wait complete. 2024-03-11T18:30:03.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.210+0000 7fbe7c600640 1 Processor -- start 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.210+0000 7fbe7c600640 1 -- start start 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.210+0000 7fbe7c600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.210+0000 7fbe7c600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 0x7fbe7419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7c600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe741a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7c600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741a5390 con 0x7fbe74109370 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7c600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741a5500 con 0x7fbe741077c0 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7c600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741a56c0 con 0x7fbe741089b0 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe741a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:33578/0 (socket says 172.21.15.2:33578) 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79c00640 1 -- 172.21.15.2:0/2618750181 learned_addr learned my addr 172.21.15.2:0/2618750181 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79200640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 0x7fbe7419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:03.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 -- 172.21.15.2:0/2618750181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 msgr2=0x7fbe7419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 0x7fbe7419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 -- 172.21.15.2:0/2618750181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 msgr2=0x7fbe7419d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe7a600640 1 -- 172.21.15.2:0/2618750181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe68009840 con 0x7fbe74109370 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79200640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 0x7fbe7419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.211+0000 7fbe79c00640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe7a600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe741a4e50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e970 tx=0x7fbe6c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c00ccc0 con 0x7fbe74109370 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe741a59a0 con 0x7fbe74109370 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe6c009e40 con 0x7fbe74109370 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c004a00 con 0x7fbe74109370 2024-03-11T18:30:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.212+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe741a5e60 con 0x7fbe74109370 2024-03-11T18:30:03.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.213+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7410eae0 con 0x7fbe74109370 2024-03-11T18:30:03.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.215+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fbe6c004b60 con 0x7fbe74109370 2024-03-11T18:30:03.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.217+0000 7fbe72000640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 0x7fbe5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:03.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.217+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbe6c014070 con 0x7fbe74109370 2024-03-11T18:30:03.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.218+0000 7fbe79c00640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 0x7fbe5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:03.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.218+0000 7fbe79c00640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 0x7fbe5c0a3ef0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe680339b0 tx=0x7fbe680333e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:03.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.219+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe6c096780 con 0x7fbe74109370 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:03.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: Deploying daemon mds.c on smithi165 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6834/2758205355,v1:172.21.15.165:6835/2758205355] up:boot 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: fsmap 6 up:standby 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:03 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: Deploying daemon mds.c on smithi165 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6834/2758205355,v1:172.21.15.165:6835/2758205355] up:boot 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: fsmap 6 up:standby 2024-03-11T18:30:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:03 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:30:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.459+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe74113450 con 0x7fbe74109370 2024-03-11T18:30:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.461+0000 7fbe72000640 1 -- 172.21.15.2:0/2618750181 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5280 (secure 0 0 0) 0x7fbe6c0056b0 con 0x7fbe74109370 2024-03-11T18:30:03.464 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:03.464 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-11T18:30:03.464 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-03-11T18:30:03.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 msgr2=0x7fbe5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 0x7fbe5c0a3ef0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbe680339b0 tx=0x7fbe680333e0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 msgr2=0x7fbe741a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe741a4e50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00e970 tx=0x7fbe6c00ed70 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 shutdown_connections 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbe5c0a1a30 0x7fbe5c0a3ef0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe74109370 0x7fbe741a4e50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbe741089b0 0x7fbe7419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 --2- 172.21.15.2:0/2618750181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe741077c0 0x7fbe7419d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 >> 172.21.15.2:0/2618750181 conn(0x7fbe74076780 msgr2=0x7fbe7410bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 shutdown_connections 2024-03-11T18:30:03.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:03.464+0000 7fbe7c600640 1 -- 172.21.15.2:0/2618750181 wait complete. 2024-03-11T18:30:03.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: Deploying daemon mds.c on smithi165 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6834/2758205355,v1:172.21.15.165:6835/2758205355] up:boot 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: fsmap 6 up:standby 2024-03-11T18:30:03.483 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:03 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:30:04.248 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:04 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2618750181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:04 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2618750181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:04.655 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 systemd[1]: Starting Ceph mds.c for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:30:04.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:04 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2618750181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:04.935 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 podman[67354]: 2024-03-11 18:30:04.559218008 +0000 UTC m=+0.018441849 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:30:04.935 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 podman[67354]: 2024-03-11 18:30:04.677380724 +0000 UTC m=+0.136604555 container create b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T18:30:04.935 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 podman[67354]: 2024-03-11 18:30:04.934394885 +0000 UTC m=+0.393618717 container init b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:05 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:05.406 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 podman[67354]: 2024-03-11 18:30:04.936945351 +0000 UTC m=+0.396169192 container start b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T18:30:05.406 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 ceph-mds[67373]: -- 172.21.15.165:0/2174197593 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eeb4d061a0 con 0x55eeb4cdcc00 2024-03-11T18:30:05.406 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: starting mds.c at 2024-03-11T18:30:05.406 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:05 smithi165 bash[67354]: b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f 2024-03-11T18:30:05.406 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:30:05 smithi165 systemd[1]: Started Ceph mds.c for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:05.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:05 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:05.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:05.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:05 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 -- 172.21.15.2:0/1978846574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 msgr2=0x7f3344104870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 --2- 172.21.15.2:0/1978846574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f3344104870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f332c009860 tx=0x7f332c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:30:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 -- 172.21.15.2:0/1978846574 shutdown_connections 2024-03-11T18:30:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 --2- 172.21.15.2:0/1978846574 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3344104db0 0x7f3344105230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 --2- 172.21.15.2:0/1978846574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f3344104870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 --2- 172.21.15.2:0/1978846574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33440ffa90 0x7f3344101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 -- 172.21.15.2:0/1978846574 >> 172.21.15.2:0/1978846574 conn(0x7f33440fb6b0 msgr2=0x7f33440fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:05.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 -- 172.21.15.2:0/1978846574 shutdown_connections 2024-03-11T18:30:05.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.720+0000 7f3349800640 1 -- 172.21.15.2:0/1978846574 wait complete. 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.721+0000 7f3349800640 1 Processor -- start 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.721+0000 7f3349800640 1 -- start start 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 0x7f334419d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 0x7f33441a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:05.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33441a52e0 con 0x7f3344102400 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33441a5450 con 0x7f3344104db0 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3349800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33441a5610 con 0x7f33440ffa90 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f333a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f333a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59588/0 (socket says 172.21.15.2:59588) 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f333a000640 1 -- 172.21.15.2:0/3458730763 learned_addr learned my addr 172.21.15.2:0/3458730763 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3342a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 0x7f334419d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.722+0000 7f3343400640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 0x7f33441a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 -- 172.21.15.2:0/3458730763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 msgr2=0x7f334419d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 0x7f334419d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 -- 172.21.15.2:0/3458730763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 msgr2=0x7f33441a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 0x7f33441a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 -- 172.21.15.2:0/3458730763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334009840 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f3342a00640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 0x7f334419d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f3343400640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 0x7f33441a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333a000640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f332c034b10 tx=0x7f332c0045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c005020 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332c02c510 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c004a60 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.723+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c009560 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.724+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c009490 con 0x7f3344102400 2024-03-11T18:30:05.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.725+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33440fd1a0 con 0x7f3344102400 2024-03-11T18:30:05.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.727+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f332c03b040 con 0x7f3344102400 2024-03-11T18:30:05.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.728+0000 7f333be00640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 0x7f33280a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:05.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.728+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f332c0eece0 con 0x7f3344102400 2024-03-11T18:30:05.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.728+0000 7f3342a00640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 0x7f33280a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:05.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.729+0000 7f3342a00640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 0x7f33280a3ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3334009fd0 tx=0x7f3334009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:05.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.731+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f332c0b83a0 con 0x7f3344102400 2024-03-11T18:30:05.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.977+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3344113180 con 0x7f3344102400 2024-03-11T18:30:05.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.979+0000 7f333be00640 1 -- 172.21.15.2:0/3458730763 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6074 (secure 0 0 0) 0x7f332c0bba60 con 0x7f3344102400 2024-03-11T18:30:05.981 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:05.981 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-11T18:30:05.981 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 8 2024-03-11T18:30:05.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 msgr2=0x7f33280a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 0x7f33280a3ef0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3334009fd0 tx=0x7f3334009500 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 msgr2=0x7f334419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f332c034b10 tx=0x7f332c0045c0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 shutdown_connections 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f33280a1a30 0x7f33280a3ef0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3344104db0 0x7f33441a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3344102400 0x7f334419dd30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 --2- 172.21.15.2:0/3458730763 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33440ffa90 0x7f334419d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 >> 172.21.15.2:0/3458730763 conn(0x7f33440fb6b0 msgr2=0x7f3344100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 shutdown_connections 2024-03-11T18:30:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:05.982+0000 7f3349800640 1 -- 172.21.15.2:0/3458730763 wait complete. 2024-03-11T18:30:06.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: Deploying daemon mds.h on smithi023 2024-03-11T18:30:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6836/3141279332,v1:172.21.15.165:6837/3141279332] up:boot 2024-03-11T18:30:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: fsmap 7 up:standby 2024-03-11T18:30:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:30:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:06 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3458730763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: Deploying daemon mds.h on smithi023 2024-03-11T18:30:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6836/3141279332,v1:172.21.15.165:6837/3141279332] up:boot 2024-03-11T18:30:06.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: fsmap 7 up:standby 2024-03-11T18:30:06.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:30:06.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:06 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3458730763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: Deploying daemon mds.h on smithi023 2024-03-11T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6836/3141279332,v1:172.21.15.165:6837/3141279332] up:boot 2024-03-11T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: fsmap 7 up:standby 2024-03-11T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:30:06.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:06 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3458730763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:07.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:07 smithi002 ceph-mon[31328]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:07.556 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:07 smithi165 ceph-mon[36634]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:07.607 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:07 smithi023 ceph-mon[36406]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:07.607 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 systemd[1]: Starting Ceph mds.h for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:30:07.867 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 podman[67437]: 2024-03-11 18:30:07.50828459 +0000 UTC m=+0.017979422 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:30:07.867 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 podman[67437]: 2024-03-11 18:30:07.611722725 +0000 UTC m=+0.121417532 container create 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T18:30:07.867 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 podman[67437]: 2024-03-11 18:30:07.865843602 +0000 UTC m=+0.375538422 container init 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T18:30:08.003 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:08.155 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 podman[67437]: 2024-03-11 18:30:07.868573825 +0000 UTC m=+0.378268641 container start 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T18:30:08.155 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 ceph-mds[67456]: -- 172.21.15.23:0/2652243974 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626fb6ca1a0 con 0x5626fb6a0800 2024-03-11T18:30:08.155 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: starting mds.h at 2024-03-11T18:30:08.155 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 bash[67437]: 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 2024-03-11T18:30:08.155 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:30:07 smithi023 systemd[1]: Started Ceph mds.h for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:08.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 -- 172.21.15.2:0/74479743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104490 msgr2=0x7f439c104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:08.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 --2- 172.21.15.2:0/74479743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104490 0x7f439c104910 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4384009900 tx=0x7f438402bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 -- 172.21.15.2:0/74479743 shutdown_connections 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 --2- 172.21.15.2:0/74479743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c104e50 0x7f439c10b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 --2- 172.21.15.2:0/74479743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104490 0x7f439c104910 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 --2- 172.21.15.2:0/74479743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c1032a0 0x7f439c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 -- 172.21.15.2:0/74479743 >> 172.21.15.2:0/74479743 conn(0x7f439c0fe830 msgr2=0x7f439c100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 -- 172.21.15.2:0/74479743 shutdown_connections 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.235+0000 7f439b200640 1 -- 172.21.15.2:0/74479743 wait complete. 2024-03-11T18:30:08.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.236+0000 7f439b200640 1 Processor -- start 2024-03-11T18:30:08.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 -- start start 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c104490 0x7f439c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104e50 0x7f439c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c1a0de0 con 0x7f439c104490 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c1a0f50 con 0x7f439c104e50 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f439b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c1a1110 con 0x7f439c1032a0 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f4399e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f4399e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:44334/0 (socket says 172.21.15.2:44334) 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f4399e00640 1 -- 172.21.15.2:0/872029495 learned_addr learned my addr 172.21.15.2:0/872029495 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:08.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.237+0000 7f4393e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c104490 0x7f439c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f439a800640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104e50 0x7f439c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 -- 172.21.15.2:0/872029495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104e50 msgr2=0x7f439c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104e50 0x7f439c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 -- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c104490 msgr2=0x7f439c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c104490 0x7f439c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 -- 172.21.15.2:0/872029495 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4384009560 con 0x7f439c1032a0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4399e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f438c00e9c0 tx=0x7f438c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c009700 con 0x7f439c1032a0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f438c00cd80 con 0x7f439c1032a0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c010430 con 0x7f439c1032a0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.238+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f439c1a13f0 con 0x7f439c1032a0 2024-03-11T18:30:08.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.239+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c1a1880 con 0x7f439c1032a0 2024-03-11T18:30:08.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.240+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c10a5c0 con 0x7f439c1032a0 2024-03-11T18:30:08.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.241+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f438c003680 con 0x7f439c1032a0 2024-03-11T18:30:08.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.243+0000 7f4392a00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 0x7f43800a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:08.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.243+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f438c014070 con 0x7f439c1032a0 2024-03-11T18:30:08.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.243+0000 7f4393e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 0x7f43800a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:08.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.244+0000 7f4393e00640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 0x7f43800a3ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4384000c00 tx=0x7f438402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:08.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.247+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f438c096a70 con 0x7f439c1032a0 2024-03-11T18:30:08.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.487+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f439c10ed00 con 0x7f439c1032a0 2024-03-11T18:30:08.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.488+0000 7f4392a00640 1 -- 172.21.15.2:0/872029495 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6862 (secure 0 0 0) 0x7f438c09a130 con 0x7f439c1032a0 2024-03-11T18:30:08.490 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:08.490 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-11T18:30:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 msgr2=0x7f43800a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 0x7f43800a3ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4384000c00 tx=0x7f438402c7e0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 msgr2=0x7f439c1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f438c00e9c0 tx=0x7f438c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 shutdown_connections 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43800a1a20 0x7f43800a3ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f439c104e50 0x7f439c1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f439c104490 0x7f439c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.490+0000 7f439b200640 1 --2- 172.21.15.2:0/872029495 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f439c1032a0 0x7f439c1992f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.491+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 >> 172.21.15.2:0/872029495 conn(0x7f439c0fe830 msgr2=0x7f439c107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.491+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 shutdown_connections 2024-03-11T18:30:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:08.491+0000 7f439b200640 1 -- 172.21.15.2:0/872029495 wait complete. 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: Deploying daemon mds.i on smithi165 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:boot 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: fsmap 8 up:standby 2024-03-11T18:30:09.317 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:30:09.318 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:09 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/872029495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: Deploying daemon mds.i on smithi165 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:boot 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: fsmap 8 up:standby 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:30:09.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:09 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/872029495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: Deploying daemon mds.i on smithi165 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:boot 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: fsmap 8 up:standby 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:30:09.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:09 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/872029495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:10.316 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:10 smithi165 ceph-mon[36634]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:10.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:10 smithi023 ceph-mon[36406]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:10.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:10 smithi002 ceph-mon[31328]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:10.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:10.655 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 podman[67672]: 2024-03-11 18:30:10.553051915 +0000 UTC m=+0.138386177 container create e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 -- 172.21.15.2:0/3408017135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e80ff360 msgr2=0x7f17e80ff780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 --2- 172.21.15.2:0/3408017135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e80ff360 0x7f17e80ff780 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009900 tx=0x7f17d802bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 -- 172.21.15.2:0/3408017135 shutdown_connections 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 --2- 172.21.15.2:0/3408017135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8100a80 0x7f17e8105320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 --2- 172.21.15.2:0/3408017135 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e81000c0 0x7f17e8100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 --2- 172.21.15.2:0/3408017135 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e80ff360 0x7f17e80ff780 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 -- 172.21.15.2:0/3408017135 >> 172.21.15.2:0/3408017135 conn(0x7f17e80facb0 msgr2=0x7f17e80fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 -- 172.21.15.2:0/3408017135 shutdown_connections 2024-03-11T18:30:10.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.741+0000 7f17e7200640 1 -- 172.21.15.2:0/3408017135 wait complete. 2024-03-11T18:30:10.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.742+0000 7f17e7200640 1 Processor -- start 2024-03-11T18:30:10.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.742+0000 7f17e7200640 1 -- start start 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.742+0000 7f17e7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.742+0000 7f17e7200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e81000c0 0x7f17e819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.742+0000 7f17e7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 0x7f17e81a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a5230 con 0x7f17e80ff360 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e7200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a53a0 con 0x7f17e81000c0 2024-03-11T18:30:10.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81a5560 con 0x7f17e8100a80 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42572/0 (socket says 172.21.15.2:42572) 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 -- 172.21.15.2:0/2155620931 learned_addr learned my addr 172.21.15.2:0/2155620931 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e6800640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 0x7f17e81a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17dd400640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e81000c0 0x7f17e819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 -- 172.21.15.2:0/2155620931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 msgr2=0x7f17e81a4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 0x7f17e81a4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 -- 172.21.15.2:0/2155620931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e81000c0 msgr2=0x7f17e819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e81000c0 0x7f17e819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e5e00640 1 -- 172.21.15.2:0/2155620931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8009560 con 0x7f17e80ff360 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.743+0000 7f17e6800640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 0x7f17e81a4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17e5e00640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f17d802c6a0 tx=0x7f17d80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:10.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80044c0 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e81a57e0 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17d80052d0 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d803e450 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.744+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e81a5d40 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.745+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8104130 con 0x7f17e80ff360 2024-03-11T18:30:10.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.747+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f17d8005440 con 0x7f17e80ff360 2024-03-11T18:30:10.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.748+0000 7f17df400640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 0x7f17cc0a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:10.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.749+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f17d80ef870 con 0x7f17e80ff360 2024-03-11T18:30:10.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.749+0000 7f17dd400640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 0x7f17cc0a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:10.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.749+0000 7f17dd400640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 0x7f17cc0a3ee0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17e819ec50 tx=0x7f17d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:10.753+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f17d80b8e80 con 0x7f17e80ff360 2024-03-11T18:30:11.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.001+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e8113100 con 0x7f17e80ff360 2024-03-11T18:30:11.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.002+0000 7f17df400640 1 -- 172.21.15.2:0/2155620931 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6862 (secure 0 0 0) 0x7f17d80bc540 con 0x7f17e80ff360 2024-03-11T18:30:11.004 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:11.004 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-11T18:30:11.004 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 9 2024-03-11T18:30:11.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 msgr2=0x7f17cc0a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 0x7f17cc0a3ee0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17e819ec50 tx=0x7f17d0005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 msgr2=0x7f17e819d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f17d802c6a0 tx=0x7f17d80040c0 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 shutdown_connections 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1a20 0x7f17cc0a3ee0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8100a80 0x7f17e81a4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e81000c0 0x7f17e819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 --2- 172.21.15.2:0/2155620931 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e80ff360 0x7f17e819d740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:11.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.005+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 >> 172.21.15.2:0/2155620931 conn(0x7f17e80facb0 msgr2=0x7f17e8101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:11.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.006+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 shutdown_connections 2024-03-11T18:30:11.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:11.006+0000 7f17e7200640 1 -- 172.21.15.2:0/2155620931 wait complete. 2024-03-11T18:30:11.155 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 podman[67672]: 2024-03-11 18:30:10.81351175 +0000 UTC m=+0.398846002 container init e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T18:30:11.155 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 podman[67672]: 2024-03-11 18:30:10.816149739 +0000 UTC m=+0.401483990 container start e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T18:30:11.155 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 ceph-mds[67690]: -- 172.21.15.165:0/4061172956 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560df374a1a0 con 0x560df3720c00 2024-03-11T18:30:11.155 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: starting mds.i at 2024-03-11T18:30:11.155 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 bash[67672]: e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 2024-03-11T18:30:11.156 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:30:10 smithi165 systemd[1]: Started Ceph mds.i for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:12.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: Deploying daemon mds.g on smithi002 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2155620931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:boot 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: fsmap 9 up:standby 2024-03-11T18:30:12.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:11 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:30:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: Deploying daemon mds.g on smithi002 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2155620931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:boot 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: fsmap 9 up:standby 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:11 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:30:12.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: Deploying daemon mds.g on smithi002 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2155620931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:boot 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: fsmap 9 up:standby 2024-03-11T18:30:12.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:11 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:30:13.027 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 -- 172.21.15.2:0/445398202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4101210 msgr2=0x7fbae4101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 --2- 172.21.15.2:0/445398202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4101210 0x7fbae4101630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009900 tx=0x7fbad802bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 -- 172.21.15.2:0/445398202 shutdown_connections 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 --2- 172.21.15.2:0/445398202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4102dc0 0x7fbae4109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 --2- 172.21.15.2:0/445398202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102400 0x7fbae4102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 --2- 172.21.15.2:0/445398202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4101210 0x7fbae4101630 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 -- 172.21.15.2:0/445398202 >> 172.21.15.2:0/445398202 conn(0x7fbae40fc7a0 msgr2=0x7fbae40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 -- 172.21.15.2:0/445398202 shutdown_connections 2024-03-11T18:30:13.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.268+0000 7fbaec600640 1 -- 172.21.15.2:0/445398202 wait complete. 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.269+0000 7fbaec600640 1 Processor -- start 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.269+0000 7fbaec600640 1 -- start start 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4101210 0x7fbae4074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 0x7fbae4073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:13.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4074210 con 0x7fbae4102400 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae4074380 con 0x7fbae4101210 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbaec600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae41a12c0 con 0x7fbae4102dc0 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbae9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbae9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42600/0 (socket says 172.21.15.2:42600) 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.270+0000 7fbae9200640 1 -- 172.21.15.2:0/1507347158 learned_addr learned my addr 172.21.15.2:0/1507347158 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9c00640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4101210 0x7fbae4074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbaea600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 0x7fbae4073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 -- 172.21.15.2:0/1507347158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 msgr2=0x7fbae4073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 0x7fbae4073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 -- 172.21.15.2:0/1507347158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4101210 msgr2=0x7fbae4074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4101210 0x7fbae4074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 -- 172.21.15.2:0/1507347158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc009840 con 0x7fbae4102400 2024-03-11T18:30:13.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbaea600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 0x7fbae4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:30:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae9200640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad000ea20 tx=0x7fbad000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.271+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0009720 con 0x7fbae4102400 2024-03-11T18:30:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.272+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad8009560 con 0x7fbae4102400 2024-03-11T18:30:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.272+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad000cd80 con 0x7fbae4102400 2024-03-11T18:30:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.272+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae41a18a0 con 0x7fbae4102400 2024-03-11T18:30:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.272+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0018630 con 0x7fbae4102400 2024-03-11T18:30:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.273+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae4108530 con 0x7fbae4102400 2024-03-11T18:30:13.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.275+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fbad0004e30 con 0x7fbae4102400 2024-03-11T18:30:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.276+0000 7fbae2000640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 0x7fbacc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.276+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbad0014070 con 0x7fbae4102400 2024-03-11T18:30:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.276+0000 7fbae9c00640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 0x7fbacc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.277+0000 7fbae9c00640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 0x7fbacc0a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005ab0 tx=0x7fbad8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:13.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.282+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbad0096bd0 con 0x7fbae4102400 2024-03-11T18:30:13.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.555+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbae410cbb0 con 0x7fbae4102400 2024-03-11T18:30:13.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.556+0000 7fbae2000640 1 -- 172.21.15.2:0/1507347158 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7658 (secure 0 0 0) 0x7fbad009a290 con 0x7fbae4102400 2024-03-11T18:30:13.558 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:13.558 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T18:30:13.559 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 10 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 msgr2=0x7fbacc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 0x7fbacc0a3ef0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005ab0 tx=0x7fbad8005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 msgr2=0x7fbae4073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad000ea20 tx=0x7fbad000ee20 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 shutdown_connections 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fbacc0a1a30 0x7fbacc0a3ef0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbae4102dc0 0x7fbae4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbae4102400 0x7fbae4073570 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 --2- 172.21.15.2:0/1507347158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae4101210 0x7fbae4074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.559+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 >> 172.21.15.2:0/1507347158 conn(0x7fbae40fc7a0 msgr2=0x7fbae4105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:13.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.560+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 shutdown_connections 2024-03-11T18:30:13.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:13.560+0000 7fbaec600640 1 -- 172.21.15.2:0/1507347158 wait complete. 2024-03-11T18:30:14.046 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:13 smithi002 ceph-mon[31328]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:14.046 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:13 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1507347158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:14.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:13 smithi165 ceph-mon[36634]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:14.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:13 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1507347158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:13 smithi023 ceph-mon[36406]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:14.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:13 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1507347158' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:14.733 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 podman[73524]: 2024-03-11 18:30:14.489135738 +0000 UTC m=+0.165380201 container create c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:30:15.002 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 podman[73524]: 2024-03-11 18:30:14.759879052 +0000 UTC m=+0.436123522 container init c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-11T18:30:15.002 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 podman[73524]: 2024-03-11 18:30:14.772977892 +0000 UTC m=+0.449222362 container start c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T18:30:15.003 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 ceph-mds[73551]: -- 172.21.15.2:0/3427969605 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fbf9d941a0 con 0x55fbf9d6ac00 2024-03-11T18:30:15.003 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g[73542]: starting mds.g at 2024-03-11T18:30:15.003 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 bash[73524]: c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 2024-03-11T18:30:15.003 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:30:14 smithi002 systemd[1]: Started Ceph mds.g for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:15.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:15.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 -- 172.21.15.2:0/2786081326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 msgr2=0x7f70dc10b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:15.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 --2- 172.21.15.2:0/2786081326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc10b870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009be0 tx=0x7f70d802bef0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 -- 172.21.15.2:0/2786081326 shutdown_connections 2024-03-11T18:30:15.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 --2- 172.21.15.2:0/2786081326 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc10b870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 --2- 172.21.15.2:0/2786081326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc104620 0x7f70dc104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 --2- 172.21.15.2:0/2786081326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc103430 0x7f70dc103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 -- 172.21.15.2:0/2786081326 >> 172.21.15.2:0/2786081326 conn(0x7f70dc0fe9e0 msgr2=0x7f70dc100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:15.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 -- 172.21.15.2:0/2786081326 shutdown_connections 2024-03-11T18:30:15.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.785+0000 7f70e4800640 1 -- 172.21.15.2:0/2786081326 wait complete. 2024-03-11T18:30:15.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.786+0000 7f70e4800640 1 Processor -- start 2024-03-11T18:30:15.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.786+0000 7f70e4800640 1 -- start start 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 0x7f70dc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc1a0ea0 con 0x7f70dc104620 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc1a1010 con 0x7f70dc103430 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e4800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc1a11d0 con 0x7f70dc104fe0 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 0x7f70dc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:15.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34446/0 (socket says 172.21.15.2:34446) 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 -- 172.21.15.2:0/3135677191 learned_addr learned my addr 172.21.15.2:0/3135677191 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e2800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 -- 172.21.15.2:0/3135677191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 msgr2=0x7f70dc1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 -- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 msgr2=0x7f70dc1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.787+0000 7f70e1e00640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 0x7f70dc1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70e1e00640 1 -- 172.21.15.2:0/3135677191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8009840 con 0x7f70dc103430 2024-03-11T18:30:15.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70e1400640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 0x7f70dc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70e1e00640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e9c0 tx=0x7f70d000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0009700 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70d000ccc0 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0010430 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc1a14b0 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.788+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc1a1970 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.789+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc10a6f0 con 0x7f70dc103430 2024-03-11T18:30:15.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.791+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f70d0003f60 con 0x7f70dc103430 2024-03-11T18:30:15.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.792+0000 7f70d6a00640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 0x7f70c40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:15.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.793+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f70d0014070 con 0x7f70dc103430 2024-03-11T18:30:15.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.793+0000 7f70e1400640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 0x7f70c40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:15.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.794+0000 7f70e1400640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 0x7f70c40a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70dc19a8c0 tx=0x7f70cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:15.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:15.795+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f70d0096980 con 0x7f70dc103430 2024-03-11T18:30:16.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.039+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70dc10ee20 con 0x7f70dc103430 2024-03-11T18:30:16.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.040+0000 7f70d6a00640 1 -- 172.21.15.2:0/3135677191 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7f70d009a040 con 0x7f70dc103430 2024-03-11T18:30:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:16.043 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T18:30:16.043 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2024-03-11T18:30:16.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.042+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 msgr2=0x7f70c40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:16.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.042+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 0x7f70c40a3f70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70dc19a8c0 tx=0x7f70cc009260 comp rx=0 tx=0).stop 2024-03-11T18:30:16.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 msgr2=0x7f70dc1993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:16.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e9c0 tx=0x7f70d000edc0 comp rx=0 tx=0).stop 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 shutdown_connections 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70c40a1ab0 0x7f70c40a3f70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70dc104fe0 0x7f70dc1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70dc104620 0x7f70dc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 --2- 172.21.15.2:0/3135677191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc103430 0x7f70dc1993b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 >> 172.21.15.2:0/3135677191 conn(0x7f70dc0fe9e0 msgr2=0x7f70dc1078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:16.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 shutdown_connections 2024-03-11T18:30:16.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:16.043+0000 7f70e4800640 1 -- 172.21.15.2:0/3135677191 wait complete. 2024-03-11T18:30:16.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:16.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.2:6838/3374917731,v1:172.21.15.2:6839/3374917731] up:boot 2024-03-11T18:30:16.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: fsmap 10 up:standby 2024-03-11T18:30:16.227 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:16.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:15 smithi002 ceph-mon[31328]: Deploying daemon mds.a on smithi002 2024-03-11T18:30:16.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:16.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.2:6838/3374917731,v1:172.21.15.2:6839/3374917731] up:boot 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: fsmap 10 up:standby 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:15 smithi165 ceph-mon[36634]: Deploying daemon mds.a on smithi002 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:16.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.2:6838/3374917731,v1:172.21.15.2:6839/3374917731] up:boot 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: fsmap 10 up:standby 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:16.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:15 smithi023 ceph-mon[36406]: Deploying daemon mds.a on smithi002 2024-03-11T18:30:17.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:16 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3135677191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:17.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:16 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3135677191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:17.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:16 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3135677191' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:18.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:18.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:17 smithi002 ceph-mon[31328]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:18.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 -- 172.21.15.2:0/767226772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870104d40 msgr2=0x7f28701051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 --2- 172.21.15.2:0/767226772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870104d40 0x7f28701051c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2868009ca0 tx=0x7f286802c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 -- 172.21.15.2:0/767226772 shutdown_connections 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 --2- 172.21.15.2:0/767226772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870104d40 0x7f28701051c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 --2- 172.21.15.2:0/767226772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870102390 0x7f2870104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 --2- 172.21.15.2:0/767226772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f2870101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 -- 172.21.15.2:0/767226772 >> 172.21.15.2:0/767226772 conn(0x7f28700fb640 msgr2=0x7f28700fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 -- 172.21.15.2:0/767226772 shutdown_connections 2024-03-11T18:30:18.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.273+0000 7f2875200640 1 -- 172.21.15.2:0/767226772 wait complete. 2024-03-11T18:30:18.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.274+0000 7f2875200640 1 Processor -- start 2024-03-11T18:30:18.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.274+0000 7f2875200640 1 -- start start 2024-03-11T18:30:18.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.274+0000 7f2875200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f2875200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 0x7f287019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f2875200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870104d40 0x7f28701a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f2875200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701a52c0 con 0x7f2870102390 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f2875200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701a5430 con 0x7f2870104d40 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f2875200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701a55f0 con 0x7f28700ffa20 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 0x7f287019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42110/0 (socket says 172.21.15.2:42110) 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 -- 172.21.15.2:0/1021533676 learned_addr learned my addr 172.21.15.2:0/1021533676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286e800640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870104d40 0x7f28701a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 -- 172.21.15.2:0/1021533676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870104d40 msgr2=0x7f28701a4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870104d40 0x7f28701a4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 -- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 msgr2=0x7f287019dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 0x7f287019dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286de00640 1 -- 172.21.15.2:0/1021533676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009900 con 0x7f28700ffa20 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.275+0000 7f286d400640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 0x7f287019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f286de00640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f286000d950 tx=0x7f286000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860009c30 con 0x7f28700ffa20 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28701a58d0 con 0x7f28700ffa20 2024-03-11T18:30:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2860009d90 con 0x7f28700ffa20 2024-03-11T18:30:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286000f7e0 con 0x7f28700ffa20 2024-03-11T18:30:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.276+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28701a5de0 con 0x7f28700ffa20 2024-03-11T18:30:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.277+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28700fd130 con 0x7f28700ffa20 2024-03-11T18:30:18.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.279+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f2860010ad0 con 0x7f28700ffa20 2024-03-11T18:30:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.281+0000 7f2866a00640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 0x7f28540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.281+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f28600ccbf0 con 0x7f28700ffa20 2024-03-11T18:30:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.281+0000 7f286d400640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 0x7f28540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:18.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.282+0000 7f286d400640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 0x7f28540a3ef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f287019ece0 tx=0x7f285c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:18.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.285+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f28600962b0 con 0x7f28700ffa20 2024-03-11T18:30:18.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:17 smithi165 ceph-mon[36634]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:18.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:17 smithi023 ceph-mon[36406]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:18.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.535+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f287002cea0 con 0x7f28700ffa20 2024-03-11T18:30:18.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.537+0000 7f2866a00640 1 -- 172.21.15.2:0/1021533676 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7f2860099970 con 0x7f28700ffa20 2024-03-11T18:30:18.539 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:18.539 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T18:30:18.539 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 11 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 msgr2=0x7f28540a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 0x7f28540a3ef0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f287019ece0 tx=0x7f285c005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 msgr2=0x7f287019d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f286000d950 tx=0x7f286000de10 comp rx=0 tx=0).stop 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 shutdown_connections 2024-03-11T18:30:18.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f28540a1a30 0x7f28540a3ef0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2870104d40 0x7f28701a4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2870102390 0x7f287019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 --2- 172.21.15.2:0/1021533676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f28700ffa20 0x7f287019d7d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.539+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 >> 172.21.15.2:0/1021533676 conn(0x7f28700fb640 msgr2=0x7f28701003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.540+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 shutdown_connections 2024-03-11T18:30:18.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:18.540+0000 7f2875200640 1 -- 172.21.15.2:0/1021533676 wait complete. 2024-03-11T18:30:19.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1021533676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:19.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:19 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1021533676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:19.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:19 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1021533676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.2:6840/2401819833,v1:172.21.15.2:6841/2401819833] up:boot 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: fsmap 11 up:standby 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.2:6840/2401819833,v1:172.21.15.2:6841/2401819833] up:boot 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: fsmap 11 up:standby 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:30:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:20.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:20.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.2:6840/2401819833,v1:172.21.15.2:6841/2401819833] up:boot 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: fsmap 11 up:standby 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T18:30:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:20.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 -- 172.21.15.2:0/38994724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe318104490 msgr2=0x7fe318104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 --2- 172.21.15.2:0/38994724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe318104490 0x7fe318104910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe300009900 tx=0x7fe30002bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 -- 172.21.15.2:0/38994724 shutdown_connections 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 --2- 172.21.15.2:0/38994724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 0x7fe31810b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 --2- 172.21.15.2:0/38994724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe318104490 0x7fe318104910 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 --2- 172.21.15.2:0/38994724 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe3181032a0 0x7fe3181036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:20.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 -- 172.21.15.2:0/38994724 >> 172.21.15.2:0/38994724 conn(0x7fe3180fe830 msgr2=0x7fe318100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:20.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.796+0000 7fe317e00640 1 -- 172.21.15.2:0/38994724 shutdown_connections 2024-03-11T18:30:20.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.797+0000 7fe317e00640 1 -- 172.21.15.2:0/38994724 wait complete. 2024-03-11T18:30:20.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.798+0000 7fe317e00640 1 Processor -- start 2024-03-11T18:30:20.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.798+0000 7fe317e00640 1 -- start start 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 0x7fe318199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 0x7fe3181a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3181a0dd0 con 0x7fe318104e50 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3181a0f40 con 0x7fe3181032a0 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3181a1100 con 0x7fe318104490 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe316a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe316a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60346/0 (socket says 172.21.15.2:60346) 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe316a00640 1 -- 172.21.15.2:0/2510531240 learned_addr learned my addr 172.21.15.2:0/2510531240 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe30e000640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 0x7fe318199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.799+0000 7fe317400640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 0x7fe3181a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 -- 172.21.15.2:0/2510531240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 msgr2=0x7fe3181992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 -- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 msgr2=0x7fe3181a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 0x7fe3181a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 -- 172.21.15.2:0/2510531240 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300009560 con 0x7fe318104490 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe316a00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30e000640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 0x7fe318199820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe300000c00 tx=0x7fe300004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300004aa0 con 0x7fe318104490 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3000052d0 con 0x7fe318104490 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3181a1380 con 0x7fe318104490 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.800+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300003e80 con 0x7fe318104490 2024-03-11T18:30:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.801+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181a18b0 con 0x7fe318104490 2024-03-11T18:30:20.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.802+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31810a5c0 con 0x7fe318104490 2024-03-11T18:30:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.804+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fe300005440 con 0x7fe318104490 2024-03-11T18:30:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.805+0000 7fe30fe00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 0x7fe2fc0a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.805+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3000ef330 con 0x7fe318104490 2024-03-11T18:30:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.805+0000 7fe316a00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 0x7fe2fc0a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.806+0000 7fe316a00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 0x7fe2fc0a3f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe308011750 tx=0x7fe3080096d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:20.809+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe3000b8940 con 0x7fe318104490 2024-03-11T18:30:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.053+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe31802cda0 con 0x7fe318104490 2024-03-11T18:30:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.054+0000 7fe30fe00640 1 -- 172.21.15.2:0/2510531240 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9236 (secure 0 0 0) 0x7fe3000bc000 con 0x7fe318104490 2024-03-11T18:30:21.056 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:21.056 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-11T18:30:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 12 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 msgr2=0x7fe2fc0a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 0x7fe2fc0a3f60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe308011750 tx=0x7fe3080096d0 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 msgr2=0x7fe318199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 0x7fe318199820 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe300000c00 tx=0x7fe300004270 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 shutdown_connections 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe2fc0a1aa0 0x7fe2fc0a3f60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe318104e50 0x7fe3181a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe318104490 0x7fe318199820 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 --2- 172.21.15.2:0/2510531240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3181032a0 0x7fe3181992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 >> 172.21.15.2:0/2510531240 conn(0x7fe3180fe830 msgr2=0x7fe318107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 shutdown_connections 2024-03-11T18:30:21.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:21.057+0000 7fe317e00640 1 -- 172.21.15.2:0/2510531240 wait complete. 2024-03-11T18:30:21.287 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:21 smithi023 ceph-mon[36406]: Deploying daemon mds.b on smithi023 2024-03-11T18:30:21.287 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 systemd[1]: Starting Ceph mds.b for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:30:21.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:21 smithi165 ceph-mon[36634]: Deploying daemon mds.b on smithi023 2024-03-11T18:30:21.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:21 smithi002 ceph-mon[31328]: Deploying daemon mds.b on smithi023 2024-03-11T18:30:21.653 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 podman[67742]: 2024-03-11 18:30:21.405931055 +0000 UTC m=+0.018271163 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:30:21.653 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 podman[67742]: 2024-03-11 18:30:21.526727294 +0000 UTC m=+0.139067382 container create 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 podman[67742]: 2024-03-11 18:30:21.762828814 +0000 UTC m=+0.375168913 container init 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 podman[67742]: 2024-03-11 18:30:21.765987604 +0000 UTC m=+0.378327694 container start 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 ceph-mds[67763]: -- 172.21.15.23:0/2315297610 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5558acca21a0 con 0x5558acc78800 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: starting mds.b at 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 bash[67742]: 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 2024-03-11T18:30:21.905 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:21 smithi023 systemd[1]: Started Ceph mds.b for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2510531240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2510531240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2510531240' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:22.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:23.078 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:23.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 -- 172.21.15.2:0/3446999761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104620 msgr2=0x7f2bac104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 --2- 172.21.15.2:0/3446999761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104620 0x7f2bac104aa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009900 tx=0x7f2b9802bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 -- 172.21.15.2:0/3446999761 shutdown_connections 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 --2- 172.21.15.2:0/3446999761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac104fe0 0x7f2bac10b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 --2- 172.21.15.2:0/3446999761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104620 0x7f2bac104aa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 --2- 172.21.15.2:0/3446999761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac103430 0x7f2bac103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 -- 172.21.15.2:0/3446999761 >> 172.21.15.2:0/3446999761 conn(0x7f2bac0fe9e0 msgr2=0x7f2bac100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 -- 172.21.15.2:0/3446999761 shutdown_connections 2024-03-11T18:30:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.282+0000 7f2bb1000640 1 -- 172.21.15.2:0/3446999761 wait complete. 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 Processor -- start 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 -- start start 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac103430 0x7f2bac199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 0x7f2bac1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac1a0f20 con 0x7f2bac103430 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac1a1090 con 0x7f2bac104fe0 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2bb1000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac1a1250 con 0x7f2bac104620 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2ba9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac103430 0x7f2bac199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.283+0000 7f2ba9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42154/0 (socket says 172.21.15.2:42154) 2024-03-11T18:30:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2baa800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 0x7f2bac1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 -- 172.21.15.2:0/1465717072 learned_addr learned my addr 172.21.15.2:0/1465717072 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 -- 172.21.15.2:0/1465717072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 msgr2=0x7f2bac1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 0x7f2bac1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 -- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac103430 msgr2=0x7f2bac199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac103430 0x7f2bac199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 -- 172.21.15.2:0/1465717072 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98009560 con 0x7f2bac104620 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2baa800640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 0x7f2bac1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.284+0000 7f2ba9400640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b98006010 tx=0x7f2b980040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.285+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980044c0 con 0x7f2bac104620 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.285+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b980052d0 con 0x7f2bac104620 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.285+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac1a14d0 con 0x7f2bac104620 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.285+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98003e10 con 0x7f2bac104620 2024-03-11T18:30:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.285+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac1a19e0 con 0x7f2bac104620 2024-03-11T18:30:23.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.286+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac10a750 con 0x7f2bac104620 2024-03-11T18:30:23.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.287+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f2b98005440 con 0x7f2bac104620 2024-03-11T18:30:23.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.289+0000 7f2ba2a00640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 0x7f2b900a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.289+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2b980ef330 con 0x7f2bac104620 2024-03-11T18:30:23.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.289+0000 7f2ba9e00640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 0x7f2b900a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.290+0000 7f2ba9e00640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 0x7f2b900a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2bac104480 tx=0x7f2b9c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:23.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.293+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b980b8940 con 0x7f2bac104620 2024-03-11T18:30:23.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:23 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:boot 2024-03-11T18:30:23.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:23 smithi165 ceph-mon[36634]: fsmap 12 up:standby 2024-03-11T18:30:23.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:30:23.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:23.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:23 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:boot 2024-03-11T18:30:23.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:23 smithi023 ceph-mon[36406]: fsmap 12 up:standby 2024-03-11T18:30:23.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:30:23.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:23 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:boot 2024-03-11T18:30:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:23 smithi002 ceph-mon[31328]: fsmap 12 up:standby 2024-03-11T18:30:23.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:30:23.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:23.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.544+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bac02cda0 con 0x7f2bac104620 2024-03-11T18:30:23.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.545+0000 7f2ba2a00640 1 -- 172.21.15.2:0/1465717072 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10028 (secure 0 0 0) 0x7f2b980bc000 con 0x7f2bac104620 2024-03-11T18:30:23.547 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:23.547 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-11T18:30:23.547 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 13 2024-03-11T18:30:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.547+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 msgr2=0x7f2b900a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.547+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 0x7f2b900a3ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2bac104480 tx=0x7f2b9c00b390 comp rx=0 tx=0).stop 2024-03-11T18:30:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.547+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 msgr2=0x7f2bac199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.547+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2b98006010 tx=0x7f2b980040c0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 shutdown_connections 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b900a1a30 0x7f2b900a3ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bac104fe0 0x7f2bac1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bac104620 0x7f2bac199970 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 --2- 172.21.15.2:0/1465717072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bac103430 0x7f2bac199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 >> 172.21.15.2:0/1465717072 conn(0x7f2bac0fe9e0 msgr2=0x7f2bac107920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 shutdown_connections 2024-03-11T18:30:23.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.548+0000 7f2bb1000640 1 -- 172.21.15.2:0/1465717072 wait complete. 2024-03-11T18:30:23.568 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 -- 172.21.15.2:0/3460378672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541089b0 msgr2=0x7fc554108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 --2- 172.21.15.2:0/3460378672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541089b0 0x7fc554108e30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009900 tx=0x7fc53c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 -- 172.21.15.2:0/3460378672 shutdown_connections 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 --2- 172.21.15.2:0/3460378672 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 0x7fc55410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 --2- 172.21.15.2:0/3460378672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541089b0 0x7fc554108e30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 --2- 172.21.15.2:0/3460378672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541077c0 0x7fc554107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 -- 172.21.15.2:0/3460378672 >> 172.21.15.2:0/3460378672 conn(0x7fc554076780 msgr2=0x7fc554078ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 -- 172.21.15.2:0/3460378672 shutdown_connections 2024-03-11T18:30:23.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.767+0000 7fc55a600640 1 -- 172.21.15.2:0/3460378672 wait complete. 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.768+0000 7fc55a600640 1 Processor -- start 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.768+0000 7fc55a600640 1 -- start start 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.768+0000 7fc55a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.768+0000 7fc55a600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc55a600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 0x7fc5541a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc55a600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541a5330 con 0x7fc5541077c0 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc55a600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541a54a0 con 0x7fc5541089b0 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc55a600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5541a5660 con 0x7fc554109370 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc553400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60368/0 (socket says 172.21.15.2:60368) 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc553400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52222/0 (socket says 172.21.15.2:52222) 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 -- 172.21.15.2:0/705531337 learned_addr learned my addr 172.21.15.2:0/705531337 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc553e00640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 0x7fc5541a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 -- 172.21.15.2:0/705531337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 msgr2=0x7fc5541a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 0x7fc5541a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 -- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 msgr2=0x7fc55419d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc54aa00640 1 -- 172.21.15.2:0/705531337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc544009490 con 0x7fc5541089b0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.769+0000 7fc553400640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc54aa00640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc53c02c6a0 tx=0x7fc53c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0044c0 con 0x7fc5541089b0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c0052d0 con 0x7fc5541089b0 2024-03-11T18:30:23.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c009560 con 0x7fc5541089b0 2024-03-11T18:30:23.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c003e80 con 0x7fc5541089b0 2024-03-11T18:30:23.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.770+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5541a5c60 con 0x7fc5541089b0 2024-03-11T18:30:23.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.771+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55410eae0 con 0x7fc5541089b0 2024-03-11T18:30:23.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.773+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc53c005440 con 0x7fc5541089b0 2024-03-11T18:30:23.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.774+0000 7fc550c00640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 0x7fc5380a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:23.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.774+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc53c0ef880 con 0x7fc5541089b0 2024-03-11T18:30:23.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.774+0000 7fc553400640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 0x7fc5380a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:23.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.775+0000 7fc553400640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 0x7fc5380a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc554108810 tx=0x7fc544005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:23.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.780+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc53c0b8fb0 con 0x7fc5541089b0 2024-03-11T18:30:23.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.982+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc55402cdd0 con 0x7fc5541089b0 2024-03-11T18:30:23.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.983+0000 7fc550c00640 1 -- 172.21.15.2:0/705531337 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+18375 (secure 0 0 0) 0x7fc53c0bc670 con 0x7fc5541089b0 2024-03-11T18:30:23.985 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:23.985 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":83,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:29:30.822013+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:23.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.985+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 msgr2=0x7fc5380a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.985+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 0x7fc5380a3ef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc554108810 tx=0x7fc544005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:23.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.985+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 msgr2=0x7fc55419dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:23.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.985+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc53c02c6a0 tx=0x7fc53c0040c0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 shutdown_connections 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5380a1a30 0x7fc5380a3ef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc554109370 0x7fc5541a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5541089b0 0x7fc55419dd80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 --2- 172.21.15.2:0/705531337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5541077c0 0x7fc55419d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 >> 172.21.15.2:0/705531337 conn(0x7fc554076780 msgr2=0x7fc55410bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 shutdown_connections 2024-03-11T18:30:23.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:23.986+0000 7fc55a600640 1 -- 172.21.15.2:0/705531337 wait complete. 2024-03-11T18:30:24.006 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T18:30:24.006 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 -- 172.21.15.2:0/1542392535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 msgr2=0x7efec4104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 --2- 172.21.15.2:0/1542392535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4104a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009900 tx=0x7efeb002bc50 comp rx=0 tx=0).stop 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 -- 172.21.15.2:0/1542392535 shutdown_connections 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 --2- 172.21.15.2:0/1542392535 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 0x7efec410b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 --2- 172.21.15.2:0/1542392535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4104a20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 --2- 172.21.15.2:0/1542392535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 0x7efec41037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 -- 172.21.15.2:0/1542392535 >> 172.21.15.2:0/1542392535 conn(0x7efec40fe940 msgr2=0x7efec4100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 -- 172.21.15.2:0/1542392535 shutdown_connections 2024-03-11T18:30:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.210+0000 7efec9200640 1 -- 172.21.15.2:0/1542392535 wait complete. 2024-03-11T18:30:24.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.211+0000 7efec9200640 1 Processor -- start 2024-03-11T18:30:24.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 -- start start 2024-03-11T18:30:24.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 0x7efec4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 0x7efec41a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec41a0f00 con 0x7efec4104f60 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec41a1070 con 0x7efec41033b0 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec9200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec41a1230 con 0x7efec41045a0 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 0x7efec41a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec1400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec1e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 0x7efec4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec1400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42198/0 (socket says 172.21.15.2:42198) 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.212+0000 7efec1400640 1 -- 172.21.15.2:0/2202908562 learned_addr learned my addr 172.21.15.2:0/2202908562 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 -- 172.21.15.2:0/2202908562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 msgr2=0x7efec4199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 0x7efec4199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 -- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 msgr2=0x7efec41a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 0x7efec41a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 -- 172.21.15.2:0/2202908562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009560 con 0x7efec41045a0 2024-03-11T18:30:24.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec1400640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efeb002c6c0 tx=0x7efeb00040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:24.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00044c0 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efeb00052d0 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.213+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec41a14b0 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.214+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb003e450 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.214+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec41a19c0 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.215+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec410a6d0 con 0x7efec41045a0 2024-03-11T18:30:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.216+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7efeb0005440 con 0x7efec41045a0 2024-03-11T18:30:24.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.217+0000 7efebaa00640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 0x7efea80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.218+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efeb00ef7d0 con 0x7efec41045a0 2024-03-11T18:30:24.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.218+0000 7efec1e00640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 0x7efea80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.218+0000 7efec1e00640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 0x7efea80a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efec4104400 tx=0x7efeb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:24.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.221+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efeb00b8e60 con 0x7efec41045a0 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1465717072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:24 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/705531337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1465717072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:24 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/705531337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.453+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efec410efc0 con 0x7efec41045a0 2024-03-11T18:30:24.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.454+0000 7efebaa00640 1 -- 172.21.15.2:0/2202908562 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7efeb00bc520 con 0x7efec41045a0 2024-03-11T18:30:24.455 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.456+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 msgr2=0x7efea80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.456+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 0x7efea80a3ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efec4104400 tx=0x7efeb4005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.456+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 msgr2=0x7efec4199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.456+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efeb002c6c0 tx=0x7efeb00040c0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 shutdown_connections 2024-03-11T18:30:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efea80a1a30 0x7efea80a3ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efec4104f60 0x7efec41a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efec41045a0 0x7efec4199950 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 --2- 172.21.15.2:0/2202908562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec41033b0 0x7efec4199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 >> 172.21.15.2:0/2202908562 conn(0x7efec40fe940 msgr2=0x7efec41078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 shutdown_connections 2024-03-11T18:30:24.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.457+0000 7efec9200640 1 -- 172.21.15.2:0/2202908562 wait complete. 2024-03-11T18:30:24.477 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:24.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1465717072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:24.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:24 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/705531337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:24.481 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 -- 172.21.15.2:0/2457733238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed88104590 msgr2=0x7fed88104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 --2- 172.21.15.2:0/2457733238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed88104590 0x7fed88104a10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed74009900 tx=0x7fed7402bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 -- 172.21.15.2:0/2457733238 shutdown_connections 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 --2- 172.21.15.2:0/2457733238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104f50 0x7fed8810b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 --2- 172.21.15.2:0/2457733238 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed88104590 0x7fed88104a10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 --2- 172.21.15.2:0/2457733238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed881033a0 0x7fed881037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 -- 172.21.15.2:0/2457733238 >> 172.21.15.2:0/2457733238 conn(0x7fed880fe930 msgr2=0x7fed88100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 -- 172.21.15.2:0/2457733238 shutdown_connections 2024-03-11T18:30:24.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.693+0000 7fed8d800640 1 -- 172.21.15.2:0/2457733238 wait complete. 2024-03-11T18:30:24.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.694+0000 7fed8d800640 1 Processor -- start 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.694+0000 7fed8d800640 1 -- start start 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 0x7fed88199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 0x7fed88199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed88104f50 0x7fed881a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed881a0ef0 con 0x7fed88104590 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed881a1060 con 0x7fed88104f50 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed8d800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed881a1220 con 0x7fed881033a0 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 0x7fed88199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 0x7fed88199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 0x7fed88199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52254/0 (socket says 172.21.15.2:52254) 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86000640 1 -- 172.21.15.2:0/2461039872 learned_addr learned my addr 172.21.15.2:0/2461039872 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed87400640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed88104f50 0x7fed881a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 -- 172.21.15.2:0/2461039872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed88104f50 msgr2=0x7fed881a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed88104f50 0x7fed881a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 -- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 msgr2=0x7fed88199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 0x7fed88199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.695+0000 7fed86a00640 1 -- 172.21.15.2:0/2461039872 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed78009490 con 0x7fed881033a0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed86a00640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 0x7fed88199400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fed7800e9c0 tx=0x7fed7800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7800ccc0 con 0x7fed881033a0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed7800ce20 con 0x7fed881033a0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80009840 con 0x7fed881033a0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed780187e0 con 0x7fed881033a0 2024-03-11T18:30:24.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.696+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed74009560 con 0x7fed881033a0 2024-03-11T18:30:24.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.697+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8810a660 con 0x7fed881033a0 2024-03-11T18:30:24.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.700+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fed78010430 con 0x7fed881033a0 2024-03-11T18:30:24.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.701+0000 7fed7f400640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 0x7fed6c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:24.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.701+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed78014070 con 0x7fed881033a0 2024-03-11T18:30:24.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.702+0000 7fed86000640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 0x7fed6c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:24.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.702+0000 7fed86000640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 0x7fed6c0a3f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fed8819a910 tx=0x7fed7402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:24.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.706+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed780969d0 con 0x7fed881033a0 2024-03-11T18:30:24.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.914+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fed8802ce40 con 0x7fed881033a0 2024-03-11T18:30:24.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.915+0000 7fed7f400640 1 -- 172.21.15.2:0/2461039872 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) v1 ==== 84+0+65 (secure 0 0 0) 0x7fed7809a090 con 0x7fed881033a0 2024-03-11T18:30:24.916 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:30:24.917 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-03-11T18:30:24.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.917+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 msgr2=0x7fed6c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.917+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 0x7fed6c0a3f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fed8819a910 tx=0x7fed7402c7e0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.917+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 msgr2=0x7fed88199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:24.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.917+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 0x7fed88199400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fed7800e9c0 tx=0x7fed7800edc0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 shutdown_connections 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fed6c0a1ab0 0x7fed6c0a3f70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed88104f50 0x7fed881a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed88104590 0x7fed88199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 --2- 172.21.15.2:0/2461039872 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed881033a0 0x7fed88199400 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 >> 172.21.15.2:0/2461039872 conn(0x7fed880fe930 msgr2=0x7fed88107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 shutdown_connections 2024-03-11T18:30:24.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:24.918+0000 7fed8d800640 1 -- 172.21.15.2:0/2461039872 wait complete. 2024-03-11T18:30:24.936 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-11T18:30:24.936 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-11T18:30:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.138+0000 7f2519800640 1 -- 172.21.15.2:0/1543729858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 msgr2=0x7f25141037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.138+0000 7f2519800640 1 --2- 172.21.15.2:0/1543729858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f25141037e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2504009900 tx=0x7f250402bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 -- 172.21.15.2:0/1543729858 shutdown_connections 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 --2- 172.21.15.2:0/1543729858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2514104f70 0x7f251410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 --2- 172.21.15.2:0/1543729858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f25141045b0 0x7f2514104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 --2- 172.21.15.2:0/1543729858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f25141037e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 -- 172.21.15.2:0/1543729858 >> 172.21.15.2:0/1543729858 conn(0x7f25140fe930 msgr2=0x7f2514100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 -- 172.21.15.2:0/1543729858 shutdown_connections 2024-03-11T18:30:25.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.139+0000 7f2519800640 1 -- 172.21.15.2:0/1543729858 wait complete. 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 Processor -- start 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 -- start start 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 0x7f2514199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2514104f70 0x7f25141a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141a0f30 con 0x7f25141045b0 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141a10a0 con 0x7f25141033c0 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2519800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141a1260 con 0x7f2514104f70 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2512a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f250be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 0x7f2514199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2512a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60422/0 (socket says 172.21.15.2:60422) 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2512a00640 1 -- 172.21.15.2:0/338676184 learned_addr learned my addr 172.21.15.2:0/338676184 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.140+0000 7f2513400640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2514104f70 0x7f25141a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 -- 172.21.15.2:0/338676184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2514104f70 msgr2=0x7f25141a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2514104f70 0x7f25141a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 -- 172.21.15.2:0/338676184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 msgr2=0x7f2514199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 -- 172.21.15.2:0/338676184 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009560 con 0x7f25141045b0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f2512a00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250be00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 0x7f2514199980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00e9f0 tx=0x7f24fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00ccc0 con 0x7f25141045b0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25141a1540 con 0x7f25141045b0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.141+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24fc00ce20 con 0x7f25141045b0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.142+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24fc004a00 con 0x7f25141045b0 2024-03-11T18:30:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.142+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25141a1a50 con 0x7f25141045b0 2024-03-11T18:30:25.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.143+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251410a6e0 con 0x7f25141045b0 2024-03-11T18:30:25.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.145+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f24fc0050b0 con 0x7f25141045b0 2024-03-11T18:30:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.146+0000 7f250b400640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 0x7f24f80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.146+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24fc014070 con 0x7f25141045b0 2024-03-11T18:30:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.147+0000 7f2512a00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 0x7f24f80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.147+0000 7f2512a00640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 0x7f24f80a3f70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2504006010 tx=0x7f250402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:25.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.152+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f24fc096510 con 0x7f25141045b0 2024-03-11T18:30:25.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:25.359+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f251402cdd0 con 0x7f25141045b0 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2202908562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2461039872' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:25.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:25.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2202908562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2461039872' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2202908562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:25.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2461039872' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-11T18:30:25.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:25.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:25.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:25.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:26.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.240+0000 7f250b400640 1 -- 172.21.15.2:0/338676184 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) v1 ==== 138+0+0 (secure 0 0 0) 0x7f24fc099bd0 con 0x7f25141045b0 2024-03-11T18:30:26.242 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-03-11T18:30:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.243+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 msgr2=0x7f24f80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.243+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 0x7f24f80a3f70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2504006010 tx=0x7f250402c7e0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.243+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 msgr2=0x7f2514199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:26.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.243+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 0x7f2514199980 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f24fc00e9f0 tx=0x7f24fc00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 shutdown_connections 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24f80a1ab0 0x7f24f80a3f70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2514104f70 0x7f25141a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25141045b0 0x7f2514199980 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 --2- 172.21.15.2:0/338676184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25141033c0 0x7f2514199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.244+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 >> 172.21.15.2:0/338676184 conn(0x7f25140fe930 msgr2=0x7f25141078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:26.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.245+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 shutdown_connections 2024-03-11T18:30:26.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.245+0000 7f2519800640 1 -- 172.21.15.2:0/338676184 wait complete. 2024-03-11T18:30:26.271 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 -- 172.21.15.2:0/1211336059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 msgr2=0x7fcce40ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 --2- 172.21.15.2:0/1211336059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce40ffc60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009900 tx=0x7fcccc02bc40 comp rx=0 tx=0).stop 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 -- 172.21.15.2:0/1211336059 shutdown_connections 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 --2- 172.21.15.2:0/1211336059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 0x7fcce4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 --2- 172.21.15.2:0/1211336059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 --2- 172.21.15.2:0/1211336059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce40ffc60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 -- 172.21.15.2:0/1211336059 >> 172.21.15.2:0/1211336059 conn(0x7fcce40fb460 msgr2=0x7fcce40fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:26.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 -- 172.21.15.2:0/1211336059 shutdown_connections 2024-03-11T18:30:26.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.459+0000 7fcce3200640 1 -- 172.21.15.2:0/1211336059 wait complete. 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.460+0000 7fcce3200640 1 Processor -- start 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.460+0000 7fcce3200640 1 -- start start 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce4199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 0x7fcce41a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce41a0d20 con 0x7fcce4100b40 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce41a0e90 con 0x7fcce40ff840 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce3200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce41a1050 con 0x7fcce41001a0 2024-03-11T18:30:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce1400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce1400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42260/0 (socket says 172.21.15.2:42260) 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce1400640 1 -- 172.21.15.2:0/1365683929 learned_addr learned my addr 172.21.15.2:0/1365683929 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce2800640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 0x7fcce41a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.461+0000 7fcce1e00640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce4199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 -- 172.21.15.2:0/1365683929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 msgr2=0x7fcce4199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce4199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 -- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 msgr2=0x7fcce41a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 0x7fcce41a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 -- 172.21.15.2:0/1365683929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd4009490 con 0x7fcce41001a0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce1400640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fccd400ea60 tx=0x7fccd400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd400cde0 con 0x7fcce41001a0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc009560 con 0x7fcce41001a0 2024-03-11T18:30:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccd40035f0 con 0x7fcce41001a0 2024-03-11T18:30:26.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd4010690 con 0x7fcce41001a0 2024-03-11T18:30:26.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.462+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce41a16b0 con 0x7fcce41001a0 2024-03-11T18:30:26.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.463+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce410a590 con 0x7fcce41001a0 2024-03-11T18:30:26.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.465+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fccd4003af0 con 0x7fcce41001a0 2024-03-11T18:30:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.466+0000 7fccdaa00640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 0x7fccc80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.466+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fccd4014070 con 0x7fcce41001a0 2024-03-11T18:30:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.467+0000 7fcce1e00640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 0x7fccc80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.467+0000 7fcce1e00640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 0x7fccc80a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000c00 tx=0x7fcccc02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.470+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fccd4096a80 con 0x7fcce41001a0 2024-03-11T18:30:26.478 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:26 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:26.478 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:26 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T18:30:26.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:26 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:26.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:26 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T18:30:26.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:26 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:30:26.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:26 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-11T18:30:26.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:26.675+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fcce410ec60 con 0x7fcce41001a0 2024-03-11T18:30:27.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.256+0000 7fccdaa00640 1 -- 172.21.15.2:0/1365683929 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) v1 ==== 171+0+0 (secure 0 0 0) 0x7fccd409a140 con 0x7fcce41001a0 2024-03-11T18:30:27.259 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-03-11T18:30:27.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 msgr2=0x7fccc80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 0x7fccc80a3ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcccc000c00 tx=0x7fcccc02c7f0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 msgr2=0x7fcce4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fccd400ea60 tx=0x7fccd400ef20 comp rx=0 tx=0).stop 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 shutdown_connections 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fccc80a1a30 0x7fccc80a3ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcce4100b40 0x7fcce41a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcce41001a0 0x7fcce4199770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 --2- 172.21.15.2:0/1365683929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcce40ff840 0x7fcce4199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.263+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 >> 172.21.15.2:0/1365683929 conn(0x7fcce40fb460 msgr2=0x7fcce40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:27.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.266+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 shutdown_connections 2024-03-11T18:30:27.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.267+0000 7fcce3200640 1 -- 172.21.15.2:0/1365683929 wait complete. 2024-03-11T18:30:27.303 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-11T18:30:27.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 -- 172.21.15.2:0/2088704114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 msgr2=0x7f37201095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 --2- 172.21.15.2:0/2088704114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201095d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3718009ca0 tx=0x7f371802c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 -- 172.21.15.2:0/2088704114 shutdown_connections 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 --2- 172.21.15.2:0/2088704114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201095d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 --2- 172.21.15.2:0/2088704114 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720102380 0x7f3720102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 --2- 172.21.15.2:0/2088704114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720101190 0x7f37201015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 -- 172.21.15.2:0/2088704114 >> 172.21.15.2:0/2088704114 conn(0x7f37200fc700 msgr2=0x7f37200feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 -- 172.21.15.2:0/2088704114 shutdown_connections 2024-03-11T18:30:27.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.486+0000 7f3727600640 1 -- 172.21.15.2:0/2088704114 wait complete. 2024-03-11T18:30:27.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.487+0000 7f3727600640 1 Processor -- start 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.487+0000 7f3727600640 1 -- start start 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720101190 0x7f37201993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37201a0ed0 con 0x7f3720102d40 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37201a1040 con 0x7f3720102380 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3727600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37201a1200 con 0x7f3720101190 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3725600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:53290/0 (socket says 172.21.15.2:53290) 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 -- 172.21.15.2:0/557358316 learned_addr learned my addr 172.21.15.2:0/557358316 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f3724c00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720101190 0x7f37201993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:27.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 -- 172.21.15.2:0/557358316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720101190 msgr2=0x7f37201993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720101190 0x7f37201993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 -- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 msgr2=0x7f37201a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.488+0000 7f371fe00640 1 -- 172.21.15.2:0/557358316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3714009490 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f371fe00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9f0 tx=0x7f370c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c009720 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f370c00cd80 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3718009900 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370c010690 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.489+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37201a1860 con 0x7f3720102380 2024-03-11T18:30:27.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.491+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372010cba0 con 0x7f3720102380 2024-03-11T18:30:27.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.493+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f370c0046c0 con 0x7f3720102380 2024-03-11T18:30:27.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.494+0000 7f371d600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 0x7f37080a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:27.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.494+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f370c014070 con 0x7f3720102380 2024-03-11T18:30:27.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.494+0000 7f3724c00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 0x7f37080a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.495+0000 7f3724c00640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 0x7f37080a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f37201021e0 tx=0x7f3714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:27.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.498+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f370c096910 con 0x7f3720102380 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1365683929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:27 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1365683929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:27 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/338676184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-11T18:30:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: osdmap e84: 12 total, 12 up, 12 in 2024-03-11T18:30:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1365683929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-11T18:30:27.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:27 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:27.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:27.787+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f37201a1d30 con 0x7f3720102380 2024-03-11T18:30:28.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.268+0000 7f371d600640 1 -- 172.21.15.2:0/557358316 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f370c099fd0 con 0x7f3720102380 2024-03-11T18:30:28.270 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 msgr2=0x7f37080a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 0x7f37080a3f70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f37201021e0 tx=0x7f3714005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 msgr2=0x7f3720199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9f0 tx=0x7f370c00edf0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 shutdown_connections 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f37080a1ab0 0x7f37080a3f70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.274+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3720102d40 0x7f37201a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.275+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3720102380 0x7f3720199920 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f370c00e9f0 tx=0x7f370c00edf0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.275+0000 7f3727600640 1 --2- 172.21.15.2:0/557358316 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3720101190 0x7f37201993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.275+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 >> 172.21.15.2:0/557358316 conn(0x7f37200fc700 msgr2=0x7f37200733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:28.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.276+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 shutdown_connections 2024-03-11T18:30:28.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.277+0000 7f3727600640 1 -- 172.21.15.2:0/557358316 wait complete. 2024-03-11T18:30:28.301 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-11T18:30:28.301 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 -- 172.21.15.2:0/3484747902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102380 msgr2=0x7fa74c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 --2- 172.21.15.2:0/3484747902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102380 0x7fa74c102800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa738009ca0 tx=0x7fa73802c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 -- 172.21.15.2:0/3484747902 shutdown_connections 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 --2- 172.21.15.2:0/3484747902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102d40 0x7fa74c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 --2- 172.21.15.2:0/3484747902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102380 0x7fa74c102800 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 --2- 172.21.15.2:0/3484747902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 -- 172.21.15.2:0/3484747902 >> 172.21.15.2:0/3484747902 conn(0x7fa74c0fc700 msgr2=0x7fa74c0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 -- 172.21.15.2:0/3484747902 shutdown_connections 2024-03-11T18:30:28.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.515+0000 7fa754200640 1 -- 172.21.15.2:0/3484747902 wait complete. 2024-03-11T18:30:28.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.516+0000 7fa754200640 1 Processor -- start 2024-03-11T18:30:28.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 -- start start 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 0x7fa74c1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 0x7fa74c1a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1a0f60 con 0x7fa74c102380 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1a10d0 con 0x7fa74c101190 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa754200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c1a1290 con 0x7fa74c102d40 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa751800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.517+0000 7fa750e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 0x7fa74c1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:53300/0 (socket says 172.21.15.2:53300) 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 -- 172.21.15.2:0/4134112703 learned_addr learned my addr 172.21.15.2:0/4134112703 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa752200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 0x7fa74c1a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 -- 172.21.15.2:0/4134112703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 msgr2=0x7fa74c1a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 0x7fa74c1a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 -- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 msgr2=0x7fa74c1999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 0x7fa74c1999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 -- 172.21.15.2:0/4134112703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009900 con 0x7fa74c101190 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa752200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 0x7fa74c1a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa750e00640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 0x7fa74c1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.518+0000 7fa751800640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e9c0 tx=0x7fa73c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.519+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c0096e0 con 0x7fa74c101190 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.519+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c1a1570 con 0x7fa74c101190 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.519+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa73c00cd80 con 0x7fa74c101190 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.519+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c010430 con 0x7fa74c101190 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.519+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c1a1a50 con 0x7fa74c101190 2024-03-11T18:30:28.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.520+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c10cba0 con 0x7fa74c101190 2024-03-11T18:30:28.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.522+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa73c003680 con 0x7fa74c101190 2024-03-11T18:30:28.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.523+0000 7fa74a000640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 0x7fa7300a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:28.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.524+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa73c014070 con 0x7fa74c101190 2024-03-11T18:30:28.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.524+0000 7fa750e00640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 0x7fa7300a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:28.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.525+0000 7fa750e00640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 0x7fa7300a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa738006010 tx=0x7fa7380326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:28.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.526+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa73c096960 con 0x7fa74c101190 2024-03-11T18:30:28.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T18:30:28.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T18:30:28.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/557358316' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/557358316' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:28.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:28.738+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fa74c10cd50 con 0x7fa74c101190 2024-03-11T18:30:28.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: osdmap e85: 12 total, 12 up, 12 in 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/557358316' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: osdmap e86: 12 total, 12 up, 12 in 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:30:28.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:30:29.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.026+0000 7fa74a000640 1 -- 172.21.15.2:0/4134112703 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) v1 ==== 157+0+0 (secure 0 0 0) 0x7fa73c09a020 con 0x7fa74c101190 2024-03-11T18:30:29.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 msgr2=0x7fa7300a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:29.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 0x7fa7300a3f70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa738006010 tx=0x7fa7380326b0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 msgr2=0x7fa74c199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:29.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e9c0 tx=0x7fa73c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 shutdown_connections 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.029+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa7300a1ab0 0x7fa7300a3f70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa74c102d40 0x7fa74c1a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74c102380 0x7fa74c1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 --2- 172.21.15.2:0/4134112703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa74c101190 0x7fa74c199470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 >> 172.21.15.2:0/4134112703 conn(0x7fa74c0fc700 msgr2=0x7fa74c0733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 shutdown_connections 2024-03-11T18:30:29.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.030+0000 7fa754200640 1 -- 172.21.15.2:0/4134112703 wait complete. 2024-03-11T18:30:29.052 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-11T18:30:29.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 -- 172.21.15.2:0/1728644337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0101c30 msgr2=0x7f9aa0104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:29.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 --2- 172.21.15.2:0/1728644337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0101c30 0x7f9aa0104020 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009ca0 tx=0x7f9a9002c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 -- 172.21.15.2:0/1728644337 shutdown_connections 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 --2- 172.21.15.2:0/1728644337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0106e90 0x7f9aa01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 --2- 172.21.15.2:0/1728644337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa0106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 --2- 172.21.15.2:0/1728644337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0101c30 0x7f9aa0104020 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 -- 172.21.15.2:0/1728644337 >> 172.21.15.2:0/1728644337 conn(0x7f9aa00fb620 msgr2=0x7f9aa00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 -- 172.21.15.2:0/1728644337 shutdown_connections 2024-03-11T18:30:29.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.288+0000 7f9aa6600640 1 -- 172.21.15.2:0/1728644337 wait complete. 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.289+0000 7f9aa6600640 1 Processor -- start 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.289+0000 7f9aa6600640 1 -- start start 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0101c30 0x7f9aa01015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa00ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:29.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa0100880 con 0x7f9aa0106e90 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa01009f0 con 0x7f9aa0101c30 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9aa6600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa01a1260 con 0x7f9aa0104560 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9a9fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9a9fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46136/0 (socket says 172.21.15.2:46136) 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9a9ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa00ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9a9fe00640 1 -- 172.21.15.2:0/3620383985 learned_addr learned my addr 172.21.15.2:0/3620383985 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.290+0000 7f9a9f400640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0101c30 0x7f9aa01015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 -- 172.21.15.2:0/3620383985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0101c30 msgr2=0x7f9aa01015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0101c30 0x7f9aa01015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 -- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 msgr2=0x7f9aa0100120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:29.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 -- 172.21.15.2:0/3620383985 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90009900 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9fe00640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a9ea00640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa00ffbe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9a88005fd0 tx=0x7f9a8800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88009c00 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a88004510 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa01a1510 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88003e70 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.291+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa01a1a20 con 0x7f9aa0104560 2024-03-11T18:30:29.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.293+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa00fd110 con 0x7f9aa0104560 2024-03-11T18:30:29.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.294+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f9a88004000 con 0x7f9aa0104560 2024-03-11T18:30:29.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.296+0000 7f9a97e00640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 0x7f9a840a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:29.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.296+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9a880ccf90 con 0x7f9aa0104560 2024-03-11T18:30:29.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.296+0000 7f9a9f400640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 0x7f9a840a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:29.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.297+0000 7f9a9f400640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 0x7f9a840a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006010 tx=0x7f9a900326b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:29.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.300+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a880961c0 con 0x7f9aa0104560 2024-03-11T18:30:29.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:29.499+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9aa002ce40 con 0x7f9aa0104560 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: pgmap v316: 97 pgs: 17 active+clean, 14 creating+peering, 66 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4134112703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T18:30:29.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:29 smithi002 ceph-mon[31328]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T18:30:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: pgmap v316: 97 pgs: 17 active+clean, 14 creating+peering, 66 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T18:30:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T18:30:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4134112703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:29 smithi023 ceph-mon[36406]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: pgmap v316: 97 pgs: 17 active+clean, 14 creating+peering, 66 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-11T18:30:29.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4134112703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T18:30:29.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-11T18:30:29.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:29 smithi165 ceph-mon[36634]: osdmap e87: 12 total, 12 up, 12 in 2024-03-11T18:30:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:30 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:active 2024-03-11T18:30:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:30 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:30 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:30 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3620383985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:30 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:30 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:active 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:30 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:30 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:30 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3620383985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:30 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:30 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] up:active 2024-03-11T18:30:31.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:30 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:30 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:31.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:30 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3620383985' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.194 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:30 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-11T18:30:31.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.289+0000 7f9a97e00640 1 -- 172.21.15.2:0/3620383985 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) v1 ==== 228+0+0 (secure 0 0 0) 0x7f9a88020100 con 0x7f9aa0104560 2024-03-11T18:30:31.293 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' created 2024-03-11T18:30:31.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.295+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 msgr2=0x7f9a840a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:31.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.295+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 0x7f9a840a3ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006010 tx=0x7f9a900326b0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.295+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 msgr2=0x7f9aa00ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.295+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa00ffbe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9a88005fd0 tx=0x7f9a8800dde0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 shutdown_connections 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a840a1a30 0x7f9a840a3ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9aa0106e90 0x7f9aa0100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9aa0104560 0x7f9aa00ffbe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 --2- 172.21.15.2:0/3620383985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9aa0101c30 0x7f9aa01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 >> 172.21.15.2:0/3620383985 conn(0x7f9aa00fb620 msgr2=0x7f9aa0102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 shutdown_connections 2024-03-11T18:30:31.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.296+0000 7f9aa6600640 1 -- 172.21.15.2:0/3620383985 wait complete. 2024-03-11T18:30:31.319 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-11T18:30:31.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.510+0000 7f693fe00640 1 -- 172.21.15.2:0/2963422218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102370 msgr2=0x7f69381027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.510+0000 7f693fe00640 1 --2- 172.21.15.2:0/2963422218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102370 0x7f69381027f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6924009900 tx=0x7f692402bc40 comp rx=0 tx=0).stop 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 -- 172.21.15.2:0/2963422218 shutdown_connections 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 --2- 172.21.15.2:0/2963422218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938102d30 0x7f69381095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 --2- 172.21.15.2:0/2963422218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102370 0x7f69381027f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 --2- 172.21.15.2:0/2963422218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938101180 0x7f69381015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 -- 172.21.15.2:0/2963422218 >> 172.21.15.2:0/2963422218 conn(0x7f69380fc6b0 msgr2=0x7f69380feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 -- 172.21.15.2:0/2963422218 shutdown_connections 2024-03-11T18:30:31.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.511+0000 7f693fe00640 1 -- 172.21.15.2:0/2963422218 wait complete. 2024-03-11T18:30:31.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.512+0000 7f693fe00640 1 Processor -- start 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.512+0000 7f693fe00640 1 -- start start 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938101180 0x7f6938199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938102370 0x7f6938199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381a0ef0 con 0x7f6938101180 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381a1060 con 0x7f6938102370 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381a1220 con 0x7f6938102d30 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938101180 0x7f6938199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:31.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938102370 0x7f6938199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34740/0 (socket says 172.21.15.2:34740) 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.513+0000 7f693de00640 1 -- 172.21.15.2:0/2097009719 learned_addr learned my addr 172.21.15.2:0/2097009719 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 -- 172.21.15.2:0/2097009719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938102370 msgr2=0x7f6938199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938102370 0x7f6938199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 -- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938101180 msgr2=0x7f6938199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938101180 0x7f6938199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 -- 172.21.15.2:0/2097009719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c009490 con 0x7f6938102d30 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693de00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f693000eaa0 tx=0x7f693000ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693000cdb0 con 0x7f6938102d30 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6930005020 con 0x7f6938102d30 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6924009560 con 0x7f6938102d30 2024-03-11T18:30:31.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69300116b0 con 0x7f6938102d30 2024-03-11T18:30:31.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.514+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69381a1850 con 0x7f6938102d30 2024-03-11T18:30:31.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.516+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693810cbe0 con 0x7f6938102d30 2024-03-11T18:30:31.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.519+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f6930005520 con 0x7f6938102d30 2024-03-11T18:30:31.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.521+0000 7f6936000640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 0x7f69200a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:31.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.521+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f69300cd720 con 0x7f6938102d30 2024-03-11T18:30:31.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.521+0000 7f693d400640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 0x7f69200a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:31.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.522+0000 7f693d400640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 0x7f69200a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69381021d0 tx=0x7f692c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:31.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.525+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69300967c0 con 0x7f6938102d30 2024-03-11T18:30:31.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:31.718+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f69381a1ad0 con 0x7f6938102d30 2024-03-11T18:30:32.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: pgmap v318: 97 pgs: 41 active+clean, 14 creating+peering, 42 unknown; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-11T18:30:32.133 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T18:30:32.133 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T18:30:32.133 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T18:30:32.133 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2097009719' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.133 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:31 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: pgmap v318: 97 pgs: 41 active+clean, 14 creating+peering, 42 unknown; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2097009719' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:31 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: pgmap v318: 97 pgs: 41 active+clean, 14 creating+peering, 42 unknown; 578 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-03-11T18:30:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: osdmap e88: 12 total, 12 up, 12 in 2024-03-11T18:30:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-11T18:30:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: osdmap e89: 12 total, 12 up, 12 in 2024-03-11T18:30:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2097009719' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:31 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-11T18:30:32.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.289+0000 7f6936000640 1 -- 172.21.15.2:0/2097009719 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) v1 ==== 169+0+0 (secure 0 0 0) 0x7f69381a1ad0 con 0x7f6938102d30 2024-03-11T18:30:32.291 INFO:teuthology.orchestra.run.smithi002.stderr:set pool 4 allow_ec_overwrites to true 2024-03-11T18:30:32.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.294+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 msgr2=0x7f69200a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:32.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.294+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 0x7f69200a3f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f69381021d0 tx=0x7f692c005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:32.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.294+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 msgr2=0x7f69381a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:32.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.294+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f693000eaa0 tx=0x7f693000ef60 comp rx=0 tx=0).stop 2024-03-11T18:30:32.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 shutdown_connections 2024-03-11T18:30:32.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69200a1ab0 0x7f69200a3f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6938102d30 0x7f69381a09b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6938102370 0x7f6938199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 --2- 172.21.15.2:0/2097009719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938101180 0x7f6938199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 >> 172.21.15.2:0/2097009719 conn(0x7f69380fc6b0 msgr2=0x7f69380733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 shutdown_connections 2024-03-11T18:30:32.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.295+0000 7f693fe00640 1 -- 172.21.15.2:0/2097009719 wait complete. 2024-03-11T18:30:32.329 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-11T18:30:32.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 -- 172.21.15.2:0/681048058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0102380 msgr2=0x7fc4c0102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/681048058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0102380 0x7fc4c0102800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009900 tx=0x7fc4a802bc40 comp rx=0 tx=0).stop 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 -- 172.21.15.2:0/681048058 shutdown_connections 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/681048058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/681048058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0102380 0x7fc4c0102800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/681048058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0101190 0x7fc4c01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 -- 172.21.15.2:0/681048058 >> 172.21.15.2:0/681048058 conn(0x7fc4c00fc700 msgr2=0x7fc4c00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 -- 172.21.15.2:0/681048058 shutdown_connections 2024-03-11T18:30:32.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.514+0000 7fc4c4e00640 1 -- 172.21.15.2:0/681048058 wait complete. 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.515+0000 7fc4c4e00640 1 Processor -- start 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.515+0000 7fc4c4e00640 1 -- start start 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.515+0000 7fc4c4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0101190 0x7fc4c01972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.515+0000 7fc4c4e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 0x7fc4c01977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.515+0000 7fc4c4e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4c4e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c019ed90 con 0x7fc4c0101190 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4c4e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c019ef00 con 0x7fc4c0102d40 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4c4e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c019f0c0 con 0x7fc4c0102380 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4be800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 0x7fc4c01977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4be800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:53346/0 (socket says 172.21.15.2:53346) 2024-03-11T18:30:32.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4be800640 1 -- 172.21.15.2:0/1587717247 learned_addr learned my addr 172.21.15.2:0/1587717247 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bde00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0101190 0x7fc4c01972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 -- 172.21.15.2:0/1587717247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 msgr2=0x7fc4c019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 -- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0101190 msgr2=0x7fc4c01972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0101190 0x7fc4c01972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4bd400640 1 -- 172.21.15.2:0/1587717247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8009560 con 0x7fc4c0102380 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.516+0000 7fc4be800640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4bd400640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 0x7fc4c01977e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc4a802c180 tx=0x7fc4a80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80044e0 con 0x7fc4c0102380 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c019f340 con 0x7fc4c0102380 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c019f870 con 0x7fc4c0102380 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4a80054e0 con 0x7fc4c0102380 2024-03-11T18:30:32.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.517+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8003e60 con 0x7fc4c0102380 2024-03-11T18:30:32.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.518+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c0068450 con 0x7fc4c0102380 2024-03-11T18:30:32.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.520+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc4a80035f0 con 0x7fc4c0102380 2024-03-11T18:30:32.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.522+0000 7fc4b6a00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 0x7fc4a40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:32.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.522+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fc4a80efdf0 con 0x7fc4c0102380 2024-03-11T18:30:32.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.522+0000 7fc4bde00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 0x7fc4a40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:32.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.523+0000 7fc4bde00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 0x7fc4a40a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005fd0 tx=0x7fc4b000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:32.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.524+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc4a80b8d80 con 0x7fc4c0102380 2024-03-11T18:30:32.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:32.767+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fc4c01084b0 con 0x7fc4c0102380 2024-03-11T18:30:33.292 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: 2024-03-11T18:30:33.030+0000 7fe52f1b5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-11T18:30:33.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.303+0000 7fc4b6a00640 1 -- 172.21.15.2:0/1587717247 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fc4a80430b0 con 0x7fc4c0102380 2024-03-11T18:30:33.305 INFO:teuthology.orchestra.run.smithi002.stderr:added data pool 4 to fsmap 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 msgr2=0x7fc4a40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 0x7fc4a40a3f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005fd0 tx=0x7fc4b000b390 comp rx=0 tx=0).stop 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 msgr2=0x7fc4c01977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 0x7fc4c01977e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc4a802c180 tx=0x7fc4a80042f0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 shutdown_connections 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a40a1ab0 0x7fc4a40a3f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c0102d40 0x7fc4c019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c0102380 0x7fc4c01977e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 --2- 172.21.15.2:0/1587717247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c0101190 0x7fc4c01972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.306+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 >> 172.21.15.2:0/1587717247 conn(0x7fc4c00fc700 msgr2=0x7fc4c0105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:33.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.307+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 shutdown_connections 2024-03-11T18:30:33.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.307+0000 7fc4c4e00640 1 -- 172.21.15.2:0/1587717247 wait complete. 2024-03-11T18:30:33.333 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:33.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 -- 172.21.15.2:0/1011647526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b40100c00 msgr2=0x7f2b40101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/1011647526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b40100c00 0x7f2b40101080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009ca0 tx=0x7f2b3802c100 comp rx=0 tx=0).stop 2024-03-11T18:30:33.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 -- 172.21.15.2:0/1011647526 shutdown_connections 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/1011647526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b40100c00 0x7f2b40101080 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/1011647526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100260 0x7f2b401006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/1011647526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b400ff900 0x7f2b400ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 -- 172.21.15.2:0/1011647526 >> 172.21.15.2:0/1011647526 conn(0x7f2b400fb500 msgr2=0x7f2b400fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 -- 172.21.15.2:0/1011647526 shutdown_connections 2024-03-11T18:30:33.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.558+0000 7f2b3fe00640 1 -- 172.21.15.2:0/1011647526 wait complete. 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.559+0000 7f2b3fe00640 1 Processor -- start 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 -- start start 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b40100260 0x7f2b40199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100c00 0x7f2b401a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b401a0e00 con 0x7f2b400ff900 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b401a0f70 con 0x7f2b40100260 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b401a1130 con 0x7f2b40100c00 2024-03-11T18:30:33.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46166/0 (socket says 172.21.15.2:46166) 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3ea00640 1 -- 172.21.15.2:0/2834308214 learned_addr learned my addr 172.21.15.2:0/2834308214 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3e000640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b40100260 0x7f2b40199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.560+0000 7f2b3f400640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100c00 0x7f2b401a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 -- 172.21.15.2:0/2834308214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100c00 msgr2=0x7f2b401a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100c00 0x7f2b401a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 -- 172.21.15.2:0/2834308214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b40100260 msgr2=0x7f2b40199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b40100260 0x7f2b40199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 -- 172.21.15.2:0/2834308214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b30009490 con 0x7f2b400ff900 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3ea00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000ea60 tx=0x7f2b3000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:33.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3000cde0 con 0x7f2b400ff900 2024-03-11T18:30:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b300035f0 con 0x7f2b400ff900 2024-03-11T18:30:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b38009900 con 0x7f2b400ff900 2024-03-11T18:30:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30010690 con 0x7f2b400ff900 2024-03-11T18:30:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.561+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b401a17e0 con 0x7f2b400ff900 2024-03-11T18:30:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.563+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4010a650 con 0x7f2b400ff900 2024-03-11T18:30:33.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.564+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f2b30003af0 con 0x7f2b400ff900 2024-03-11T18:30:33.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.566+0000 7f2b37400640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 0x7f2b240a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:33.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.566+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2b30014070 con 0x7f2b400ff900 2024-03-11T18:30:33.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.566+0000 7f2b3e000640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 0x7f2b240a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:33.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.567+0000 7f2b3e000640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 0x7f2b240a3f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b4019a820 tx=0x7f2b28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.571+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2b300969c0 con 0x7f2b400ff900 2024-03-11T18:30:33.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: pgmap v321: 129 pgs: 87 active+clean, 15 creating+peering, 27 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1587717247' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T18:30:33.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:33 smithi165 ceph-mon[36634]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: pgmap v321: 129 pgs: 87 active+clean, 15 creating+peering, 27 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1587717247' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T18:30:33.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:33 smithi023 ceph-mon[36406]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T18:30:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: pgmap v321: 129 pgs: 87 active+clean, 15 creating+peering, 27 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: osdmap e90: 12 total, 12 up, 12 in 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1587717247' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T18:30:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:33 smithi002 ceph-mon[31328]: osdmap e91: 12 total, 12 up, 12 in 2024-03-11T18:30:33.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.809+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b4010ecd0 con 0x7f2b400ff900 2024-03-11T18:30:33.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.811+0000 7f2b37400640 1 -- 172.21.15.2:0/2834308214 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11271 (secure 0 0 0) 0x7f2b3009a080 con 0x7f2b400ff900 2024-03-11T18:30:33.814 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:33.814 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:30:32.770453+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-11T18:30:33.814 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2024-03-11T18:30:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 msgr2=0x7f2b240a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 0x7f2b240a3f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b4019a820 tx=0x7f2b28005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 msgr2=0x7f2b40199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2b3000ea60 tx=0x7f2b3000ef20 comp rx=0 tx=0).stop 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 shutdown_connections 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f2b240a1aa0 0x7f2b240a3f60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b40100c00 0x7f2b401a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b40100260 0x7f2b40199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 --2- 172.21.15.2:0/2834308214 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b400ff900 0x7f2b40199310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 >> 172.21.15.2:0/2834308214 conn(0x7f2b400fb500 msgr2=0x7f2b400fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 shutdown_connections 2024-03-11T18:30:33.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:33.814+0000 7f2b3fe00640 1 -- 172.21.15.2:0/2834308214 wait complete. 2024-03-11T18:30:33.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 -- 172.21.15.2:0/3823356939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9458104e70 msgr2=0x7f945810b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 --2- 172.21.15.2:0/3823356939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9458104e70 0x7f945810b700 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9450009be0 tx=0x7f945002bef0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 -- 172.21.15.2:0/3823356939 shutdown_connections 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 --2- 172.21.15.2:0/3823356939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9458104e70 0x7f945810b700 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 --2- 172.21.15.2:0/3823356939 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 0x7f9458104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 --2- 172.21.15.2:0/3823356939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f94581032c0 0x7f94581036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 -- 172.21.15.2:0/3823356939 >> 172.21.15.2:0/3823356939 conn(0x7f94580fe830 msgr2=0x7f9458100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:34.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 -- 172.21.15.2:0/3823356939 shutdown_connections 2024-03-11T18:30:34.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.066+0000 7f945ea00640 1 -- 172.21.15.2:0/3823356939 wait complete. 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.067+0000 7f945ea00640 1 Processor -- start 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 -- start start 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 0x7f9458199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 0x7f9458199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581a0e00 con 0x7f9458104e70 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581a0f70 con 0x7f94581032c0 2024-03-11T18:30:34.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945ea00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94581a1130 con 0x7f94581044b0 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46182/0 (socket says 172.21.15.2:46182) 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945e000640 1 -- 172.21.15.2:0/3157818329 learned_addr learned my addr 172.21.15.2:0/3157818329 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945d600640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 0x7f9458199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.068+0000 7f945cc00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 0x7f9458199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 -- 172.21.15.2:0/3157818329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 msgr2=0x7f9458199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 0x7f9458199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 -- 172.21.15.2:0/3157818329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 msgr2=0x7f9458199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 0x7f9458199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 -- 172.21.15.2:0/3157818329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9448009560 con 0x7f9458104e70 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945d600640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 0x7f9458199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945e000640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9450033420 tx=0x7f9450033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9450003b80 con 0x7f9458104e70 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9450003ce0 con 0x7f9458104e70 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9450009840 con 0x7f9458104e70 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.069+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94500348c0 con 0x7f9458104e70 2024-03-11T18:30:34.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.070+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94581a1720 con 0x7f9458104e70 2024-03-11T18:30:34.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.071+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945810a580 con 0x7f9458104e70 2024-03-11T18:30:34.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.073+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f9450034a20 con 0x7f9458104e70 2024-03-11T18:30:34.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.074+0000 7f9456000640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 0x7f94400a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.074+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f94500efd30 con 0x7f9458104e70 2024-03-11T18:30:34.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.074+0000 7f945d600640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 0x7f94400a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.075+0000 7f945d600640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 0x7f94400a3f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9448009530 tx=0x7f9448009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:34.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.078+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f94500b8d50 con 0x7f9458104e70 2024-03-11T18:30:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.283+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f945802cda0 con 0x7f9458104e70 2024-03-11T18:30:34.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.285+0000 7f9456000640 1 -- 172.21.15.2:0/3157818329 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f94500dc070 con 0x7f9458104e70 2024-03-11T18:30:34.288 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:34.288 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 msgr2=0x7f94400a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 0x7f94400a3f60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9448009530 tx=0x7f9448009260 comp rx=0 tx=0).stop 2024-03-11T18:30:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 msgr2=0x7f94581a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9450033420 tx=0x7f9450033cb0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 shutdown_connections 2024-03-11T18:30:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f94400a1aa0 0x7f94400a3f60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9458104e70 0x7f94581a08c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f94581044b0 0x7f9458199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 --2- 172.21.15.2:0/3157818329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94581032c0 0x7f9458199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 >> 172.21.15.2:0/3157818329 conn(0x7f94580fe830 msgr2=0x7f9458107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.288+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 shutdown_connections 2024-03-11T18:30:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.289+0000 7f945ea00640 1 -- 172.21.15.2:0/3157818329 wait complete. 2024-03-11T18:30:34.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:34.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 -- 172.21.15.2:0/1694173020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 msgr2=0x7fc34810fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 --2- 172.21.15.2:0/1694173020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc34810fae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc344009be0 tx=0x7fc34402bef0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 -- 172.21.15.2:0/1694173020 shutdown_connections 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 --2- 172.21.15.2:0/1694173020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc34810fae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 --2- 172.21.15.2:0/1694173020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc348108890 0x7fc348108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 --2- 172.21.15.2:0/1694173020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3481076a0 0x7fc348107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 -- 172.21.15.2:0/1694173020 >> 172.21.15.2:0/1694173020 conn(0x7fc348076640 msgr2=0x7fc348078a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 -- 172.21.15.2:0/1694173020 shutdown_connections 2024-03-11T18:30:34.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.529+0000 7fc34f200640 1 -- 172.21.15.2:0/1694173020 wait complete. 2024-03-11T18:30:34.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 Processor -- start 2024-03-11T18:30:34.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 -- start start 2024-03-11T18:30:34.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc348108890 0x7fc34819dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc3481a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3481a5150 con 0x7fc3481076a0 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3481a52c0 con 0x7fc348109250 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.530+0000 7fc34f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3481a5480 con 0x7fc348108890 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46190/0 (socket says 172.21.15.2:46190) 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 -- 172.21.15.2:0/2833763433 learned_addr learned my addr 172.21.15.2:0/2833763433 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34e800640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc3481a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34d400640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc348108890 0x7fc34819dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 -- 172.21.15.2:0/2833763433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc348108890 msgr2=0x7fc34819dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc348108890 0x7fc34819dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 -- 172.21.15.2:0/2833763433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 msgr2=0x7fc3481a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc3481a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 -- 172.21.15.2:0/2833763433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338009560 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.531+0000 7fc34de00640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e9c0 tx=0x7fc34000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.532+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400096e0 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.532+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc344009840 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.532+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3481a5ae0 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.532+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc34000cd80 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.532+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340010430 con 0x7fc3481076a0 2024-03-11T18:30:34.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.533+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c0052c0 con 0x7fc3481076a0 2024-03-11T18:30:34.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.535+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc3400035f0 con 0x7fc3481076a0 2024-03-11T18:30:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.536+0000 7fc33ea00640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 0x7fc3340a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.536+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc340014070 con 0x7fc3481076a0 2024-03-11T18:30:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.537+0000 7fc34d400640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 0x7fc3340a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.537+0000 7fc34d400640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 0x7fc3340a3ee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc34819eb70 tx=0x7fc338009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:34.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.539+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc340096a80 con 0x7fc3481076a0 2024-03-11T18:30:34.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:34 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T18:30:34.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:34 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:34.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:34 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2834308214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:34 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3157818329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:34 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T18:30:34.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:34 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:34.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:34 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2834308214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:34 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3157818329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.740+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc32c005840 con 0x7fc3481076a0 2024-03-11T18:30:34.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.743+0000 7fc33ea00640 1 -- 172.21.15.2:0/2833763433 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7fc34009a140 con 0x7fc3481076a0 2024-03-11T18:30:34.746 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:34.746 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:34.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 msgr2=0x7fc3340a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 0x7fc3340a3ee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc34819eb70 tx=0x7fc338009260 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 msgr2=0x7fc34819d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc34000e9c0 tx=0x7fc34000ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 shutdown_connections 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc3340a1a20 0x7fc3340a3ee0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc348109250 0x7fc3481a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc348108890 0x7fc34819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 --2- 172.21.15.2:0/2833763433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3481076a0 0x7fc34819d660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 >> 172.21.15.2:0/2833763433 conn(0x7fc348076640 msgr2=0x7fc34810bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 shutdown_connections 2024-03-11T18:30:34.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:34.746+0000 7fc34f200640 1 -- 172.21.15.2:0/2833763433 wait complete. 2024-03-11T18:30:34.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:34 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-11T18:30:34.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:34 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:34.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:34 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2834308214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:34 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3157818329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:34.784 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:30:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 -- 172.21.15.2:0/37881908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558109050 msgr2=0x7fd55810b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 --2- 172.21.15.2:0/37881908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558109050 0x7fd55810b4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd550009ca0 tx=0x7fd55002c180 comp rx=0 tx=0).stop 2024-03-11T18:30:35.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 -- 172.21.15.2:0/37881908 shutdown_connections 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 --2- 172.21.15.2:0/37881908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558109050 0x7fd55810b4f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 --2- 172.21.15.2:0/37881908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558073b80 0x7fd558073fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 --2- 172.21.15.2:0/37881908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd5580735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.014+0000 7fd55da00640 1 -- 172.21.15.2:0/37881908 >> 172.21.15.2:0/37881908 conn(0x7fd5580fc710 msgr2=0x7fd5580feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.015+0000 7fd55da00640 1 -- 172.21.15.2:0/37881908 shutdown_connections 2024-03-11T18:30:35.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.015+0000 7fd55da00640 1 -- 172.21.15.2:0/37881908 wait complete. 2024-03-11T18:30:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 Processor -- start 2024-03-11T18:30:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 -- start start 2024-03-11T18:30:35.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558073b80 0x7fd5581993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558109050 0x7fd5581a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581a0eb0 con 0x7fd558075160 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581a1020 con 0x7fd558073b80 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd55da00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581a11e0 con 0x7fd558109050 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd556000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd556000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46222/0 (socket says 172.21.15.2:46222) 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd556000640 1 -- 172.21.15.2:0/1596974904 learned_addr learned my addr 172.21.15.2:0/1596974904 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd556a00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558073b80 0x7fd5581993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.016+0000 7fd557400640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558109050 0x7fd5581a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 -- 172.21.15.2:0/1596974904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558109050 msgr2=0x7fd5581a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558109050 0x7fd5581a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 -- 172.21.15.2:0/1596974904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558073b80 msgr2=0x7fd5581993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558073b80 0x7fd5581993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 -- 172.21.15.2:0/1596974904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd550009900 con 0x7fd558075160 2024-03-11T18:30:35.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd556000640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd54400e9f0 tx=0x7fd54400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.017+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544009700 con 0x7fd558075160 2024-03-11T18:30:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.018+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5581a14c0 con 0x7fd558075160 2024-03-11T18:30:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.018+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd54400cd80 con 0x7fd558075160 2024-03-11T18:30:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.018+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd544018920 con 0x7fd558075160 2024-03-11T18:30:35.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.018+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5581a1a20 con 0x7fd558075160 2024-03-11T18:30:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.019+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd558107e10 con 0x7fd558075160 2024-03-11T18:30:35.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.022+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd544018a80 con 0x7fd558075160 2024-03-11T18:30:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.023+0000 7fd54f400640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 0x7fd53c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.023+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd544014070 con 0x7fd558075160 2024-03-11T18:30:35.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.023+0000 7fd556a00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 0x7fd53c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.024+0000 7fd556a00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 0x7fd53c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd548009bd0 tx=0x7fd548005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.025+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd544096ec0 con 0x7fd558075160 2024-03-11T18:30:35.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.269+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd55810edd0 con 0x7fd558075160 2024-03-11T18:30:35.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.271+0000 7fd54f400640 1 -- 172.21.15.2:0/1596974904 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11271 (secure 0 0 0) 0x7fd544026090 con 0x7fd558075160 2024-03-11T18:30:35.274 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:35.274 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:30:32.770453+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-11T18:30:35.274 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 17 2024-03-11T18:30:35.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 msgr2=0x7fd53c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 0x7fd53c0a3ef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd548009bd0 tx=0x7fd548005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:35.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 msgr2=0x7fd558199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd54400e9f0 tx=0x7fd54400eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 shutdown_connections 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd53c0a1a30 0x7fd53c0a3ef0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd558109050 0x7fd5581a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd558075160 0x7fd558199900 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 --2- 172.21.15.2:0/1596974904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd558073b80 0x7fd5581993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 >> 172.21.15.2:0/1596974904 conn(0x7fd5580fc710 msgr2=0x7fd558104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 shutdown_connections 2024-03-11T18:30:35.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.274+0000 7fd55da00640 1 -- 172.21.15.2:0/1596974904 wait complete. 2024-03-11T18:30:35.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 -- 172.21.15.2:0/730805632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c104620 msgr2=0x7fd27c104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 --2- 172.21.15.2:0/730805632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c104620 0x7fd27c104aa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd268009900 tx=0x7fd26802bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 -- 172.21.15.2:0/730805632 shutdown_connections 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 --2- 172.21.15.2:0/730805632 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c10b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 --2- 172.21.15.2:0/730805632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c104620 0x7fd27c104aa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 --2- 172.21.15.2:0/730805632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c103430 0x7fd27c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 -- 172.21.15.2:0/730805632 >> 172.21.15.2:0/730805632 conn(0x7fd27c0fe9e0 msgr2=0x7fd27c100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 -- 172.21.15.2:0/730805632 shutdown_connections 2024-03-11T18:30:35.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.520+0000 7fd281a00640 1 -- 172.21.15.2:0/730805632 wait complete. 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.521+0000 7fd281a00640 1 Processor -- start 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 -- start start 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c104620 0x7fd27c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c1a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c1a0f70 con 0x7fd27c103430 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c1a10e0 con 0x7fd27c104620 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd281a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c1a12a0 con 0x7fd27c104fe0 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd27aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd27aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46246/0 (socket says 172.21.15.2:46246) 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd27aa00640 1 -- 172.21.15.2:0/3784916137 learned_addr learned my addr 172.21.15.2:0/3784916137 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd27b400640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c1a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.522+0000 7fd27a000640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c104620 0x7fd27c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 -- 172.21.15.2:0/3784916137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 msgr2=0x7fd27c1a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c1a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 -- 172.21.15.2:0/3784916137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c104620 msgr2=0x7fd27c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c104620 0x7fd27c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 -- 172.21.15.2:0/3784916137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd268009560 con 0x7fd27c103430 2024-03-11T18:30:35.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27b400640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd27aa00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9c0 tx=0x7fd26c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c009700 con 0x7fd27c103430 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd26c00cd80 con 0x7fd27c103430 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c010430 con 0x7fd27c103430 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.523+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c1a1580 con 0x7fd27c103430 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.524+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c1a1a10 con 0x7fd27c103430 2024-03-11T18:30:35.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.525+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c10a6f0 con 0x7fd27c103430 2024-03-11T18:30:35.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.527+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd26c003680 con 0x7fd27c103430 2024-03-11T18:30:35.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.528+0000 7fd273400640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 0x7fd2600a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.528+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd26c014070 con 0x7fd27c103430 2024-03-11T18:30:35.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.528+0000 7fd27a000640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 0x7fd2600a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.529+0000 7fd27a000640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 0x7fd2600a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd26802c740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.532+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd26c096920 con 0x7fd27c103430 2024-03-11T18:30:35.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:35 smithi023 ceph-mon[36406]: pgmap v324: 129 pgs: 106 active+clean, 5 creating+peering, 18 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-11T18:30:35.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:35 smithi023 ceph-mon[36406]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T18:30:35.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:35 smithi023 ceph-mon[36406]: Cluster is now healthy 2024-03-11T18:30:35.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:35 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2833763433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:35 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1596974904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:35 smithi165 ceph-mon[36634]: pgmap v324: 129 pgs: 106 active+clean, 5 creating+peering, 18 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-11T18:30:35.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:35 smithi165 ceph-mon[36634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T18:30:35.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:35 smithi165 ceph-mon[36634]: Cluster is now healthy 2024-03-11T18:30:35.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:35 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2833763433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:35 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1596974904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.726+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd27c10ee20 con 0x7fd27c103430 2024-03-11T18:30:35.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.729+0000 7fd273400640 1 -- 172.21.15.2:0/3784916137 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7fd26c099fe0 con 0x7fd27c103430 2024-03-11T18:30:35.731 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:35.732 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 msgr2=0x7fd2600a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 0x7fd2600a3f70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd268000c00 tx=0x7fd26802c740 comp rx=0 tx=0).stop 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 msgr2=0x7fd27c199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00e9c0 tx=0x7fd26c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 shutdown_connections 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2600a1ab0 0x7fd2600a3f70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd27c104fe0 0x7fd27c1a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd27c104620 0x7fd27c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 --2- 172.21.15.2:0/3784916137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd27c103430 0x7fd27c199480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 >> 172.21.15.2:0/3784916137 conn(0x7fd27c0fe9e0 msgr2=0x7fd27c1078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 shutdown_connections 2024-03-11T18:30:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.731+0000 7fd281a00640 1 -- 172.21.15.2:0/3784916137 wait complete. 2024-03-11T18:30:35.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:35 smithi002 ceph-mon[31328]: pgmap v324: 129 pgs: 106 active+clean, 5 creating+peering, 18 unknown; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-03-11T18:30:35.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:35 smithi002 ceph-mon[31328]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T18:30:35.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:35 smithi002 ceph-mon[31328]: Cluster is now healthy 2024-03-11T18:30:35.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:35 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2833763433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:35 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1596974904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:30:35.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-11T18:30:27.790111+0000', 'modified': '2024-03-11T18:30:32.770453+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.23:6839/2250151518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 2250151518}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 2250151518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-11T18:30:35.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-11T18:30:35.752 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T18:30:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.969+0000 7fb51e800640 1 -- 172.21.15.2:0/2131948936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181031e0 msgr2=0x7fb518103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.969+0000 7fb51e800640 1 --2- 172.21.15.2:0/2131948936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181031e0 0x7fb518103600 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009900 tx=0x7fb50c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.969+0000 7fb51e800640 1 -- 172.21.15.2:0/2131948936 shutdown_connections 2024-03-11T18:30:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 --2- 172.21.15.2:0/2131948936 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb518104d90 0x7fb51810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 --2- 172.21.15.2:0/2131948936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb5181043d0 0x7fb518104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 --2- 172.21.15.2:0/2131948936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181031e0 0x7fb518103600 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 -- 172.21.15.2:0/2131948936 >> 172.21.15.2:0/2131948936 conn(0x7fb5180fe750 msgr2=0x7fb518100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 -- 172.21.15.2:0/2131948936 shutdown_connections 2024-03-11T18:30:35.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.970+0000 7fb51e800640 1 -- 172.21.15.2:0/2131948936 wait complete. 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 Processor -- start 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 -- start start 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5181031e0 0x7fb5181992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181043d0 0x7fb518199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181a0db0 con 0x7fb518104d90 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181a0f20 con 0x7fb5181043d0 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51e800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5181a10e0 con 0x7fb5181031e0 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46250/0 (socket says 172.21.15.2:46250) 2024-03-11T18:30:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51de00640 1 -- 172.21.15.2:0/763555941 learned_addr learned my addr 172.21.15.2:0/763555941 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51ca00640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181043d0 0x7fb518199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.971+0000 7fb51d400640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5181031e0 0x7fb5181992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 -- 172.21.15.2:0/763555941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5181031e0 msgr2=0x7fb5181992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5181031e0 0x7fb5181992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 -- 172.21.15.2:0/763555941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181043d0 msgr2=0x7fb518199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181043d0 0x7fb518199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 -- 172.21.15.2:0/763555941 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb508009490 con 0x7fb518104d90 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51de00640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb51000eb10 tx=0x7fb51000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb516000640 1 -- 172.21.15.2:0/763555941 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510004b20 con 0x7fb518104d90 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50c009560 con 0x7fb518104d90 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb516000640 1 -- 172.21.15.2:0/763555941 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb510004c80 con 0x7fb518104d90 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb516000640 1 -- 172.21.15.2:0/763555941 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb510010670 con 0x7fb518104d90 2024-03-11T18:30:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.972+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5181a1740 con 0x7fb518104d90 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.975+0000 7fb522a6a740 1 Processor -- start 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.975+0000 7fb516000640 1 -- 172.21.15.2:0/763555941 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fb510005020 con 0x7fb518104d90 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 -- start start 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9bd2c0 0x562feb9bf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb926b30 con 0x562feb9bd2c0 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb924060 con 0x562feb9b8150 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb522a6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb91f840 con 0x562feb9b7760 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb51ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb51ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34838/0 (socket says 172.21.15.2:34838) 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb51ca00640 1 -- 172.21.15.2:0/3880420541 learned_addr learned my addr 172.21.15.2:0/3880420541 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb51ca00640 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9bd2c0 0x562feb9bf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.976+0000 7fb51d400640 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 -- 172.21.15.2:0/3880420541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 msgr2=0x562feb9bcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9bcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 -- 172.21.15.2:0/3880420541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9bd2c0 msgr2=0x562feb9bf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9bd2c0 0x562feb9bf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 -- 172.21.15.2:0/3880420541 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562feb8e74c0 con 0x562feb9b7760 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb51ca00640 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb508011990 tx=0x7fb508009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aebf265dd3e5f19c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.977+0000 7fb516000640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 0x7fb5000a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb506600640 1 -- 172.21.15.2:0/3880420541 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb508011020 con 0x562feb9b7760 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb516000640 1 -- 172.21.15.2:0/763555941 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb510014070 con 0x7fb518104d90 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb506600640 1 -- 172.21.15.2:0/3880420541 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb508011180 con 0x562feb9b7760 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb506600640 1 -- 172.21.15.2:0/3880420541 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5080035f0 con 0x562feb9b7760 2024-03-11T18:30:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb51d400640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 0x7fb5000a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb522a6a740 1 -- 172.21.15.2:0/3880420541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 msgr2=0x562feb9b7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb51d400640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 0x7fb5000a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb50c02c240 tx=0x7fb50c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.978+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb508011990 tx=0x7fb508009370 comp rx=0 tx=0).stop 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 -- 172.21.15.2:0/3880420541 shutdown_connections 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9bd2c0 0x562feb9bf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9bcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3880420541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9b7b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 -- 172.21.15.2:0/3880420541 >> 172.21.15.2:0/3880420541 conn(0x562feb9b2cd0 msgr2=0x562feb9b30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 -- 172.21.15.2:0/3880420541 shutdown_connections 2024-03-11T18:30:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 -- 172.21.15.2:0/3880420541 wait complete. 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 -- start start 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.979+0000 7fb522a6a740 1 Processor -- start 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b8150 0x562feb9c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb8c4ea0 con 0x562feb9b7760 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb8c4b40 con 0x562feb9bd2c0 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb667f20 con 0x562feb9b8150 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb522a6a740 10 client.0 _ll_register_callbacks cb 0x562feb8d8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46268/0 (socket says 172.21.15.2:46268) 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51d400640 1 -- 172.21.15.2:0/3914586961 learned_addr learned my addr 172.21.15.2:0/3914586961 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51ca00640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b8150 0x562feb9c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 -- 172.21.15.2:0/3914586961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b8150 msgr2=0x562feb9c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b8150 0x562feb9c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 -- 172.21.15.2:0/3914586961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 msgr2=0x562feb9c1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.980+0000 7fb51de00640 1 -- 172.21.15.2:0/3914586961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb508019040 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb51de00640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9c8b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5100cb440 tx=0x7fb51009afb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a6338ca46e8231 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb4faa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5100a3050 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb51009bbe0 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562feb943c40 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb51009bde0 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb522a6a740 10 client.24683 fetch_fsmap learned FSMap version 17 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.981+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562feb91dec0 con 0x562feb9bd2c0 2024-03-11T18:30:35.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562feb943c40 con 0x562feb9bd2c0 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb51001f060 con 0x562feb9bd2c0 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 10 client.24683 fetch_fsmap finished waiting for FSMap version 17 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 10 client.24683 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 20 client.24683 populate_metadata read hostname 'smithi002' 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feb9cdb10 0x562feb9d1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 4 client.24683 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562feb61f6c0 con 0x562feb9cdb10 2024-03-11T18:30:35.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.982+0000 7fb51d400640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feb9cdb10 0x562feb9d1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.983+0000 7fb51d400640 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feb9cdb10 0x562feb9d1f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d65c61a0bcad07e9 server_cookie=7a2696270dff21d0 in_seq=0 out_seq=0 2024-03-11T18:30:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.983+0000 7fb4faa00640 10 client.24683 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:30:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb4faa00640 1 -- 172.21.15.2:0/3914586961 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562feb61f6c0 con 0x562feb9cdb10 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb4faa00640 10 client.24683 handle_command_reply: tid=0 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 client.24683 shutdown 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 msgr2=0x562feb9c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9c8b90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5100cb440 tx=0x7fb51009afb0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 shutdown_connections 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feb9cdb10 0x562feb9d1f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9c8b90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b8150 0x562feb9c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb4faa00640 0 client.24683 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 --2- 172.21.15.2:0/3914586961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 >> 172.21.15.2:0/3914586961 conn(0x562feb9b2cd0 msgr2=0x562feb9b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 shutdown_connections 2024-03-11T18:30:35.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 1 -- 172.21.15.2:0/3914586961 wait complete. 2024-03-11T18:30:35.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:35.984+0000 7fb522a6a740 20 client.24683 trim_cache size 0 max 16384 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.001+0000 7fb522a6a740 1 Processor -- start 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.001+0000 7fb522a6a740 1 -- start start 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.001+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9c1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.001+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.001+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb522a6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb9c2570 con 0x562feb9b8150 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb522a6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb9cfaf0 con 0x562feb9bd2c0 2024-03-11T18:30:36.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb522a6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb61f6c0 con 0x562feb9b7760 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46276/0 (socket says 172.21.15.2:46276) 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 -- 172.21.15.2:0/1846013676 learned_addr learned my addr 172.21.15.2:0/1846013676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51d400640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9c1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51de00640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 -- 172.21.15.2:0/1846013676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 msgr2=0x562feb9c1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:36.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9c1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 -- 172.21.15.2:0/1846013676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 msgr2=0x562feb9b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51ca00640 1 -- 172.21.15.2:0/1846013676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562feb8ba8b0 con 0x562feb9b8150 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.002+0000 7fb51de00640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.003+0000 7fb51ca00640 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb50801c000 tx=0x7fb508011760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cd76d611358c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.003+0000 7fb505c00640 1 -- 172.21.15.2:0/1846013676 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb508024070 con 0x562feb9b8150 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.003+0000 7fb505c00640 1 -- 172.21.15.2:0/1846013676 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb508025040 con 0x562feb9b8150 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.003+0000 7fb505c00640 1 -- 172.21.15.2:0/1846013676 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50800fc50 con 0x562feb9b8150 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- 172.21.15.2:0/1846013676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 msgr2=0x562feb9c8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb50801c000 tx=0x7fb508011760 comp rx=0 tx=0).stop 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- 172.21.15.2:0/1846013676 shutdown_connections 2024-03-11T18:30:36.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9bd2c0 0x562feb9b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b8150 0x562feb9c8b50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 --2- 172.21.15.2:0/1846013676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9b7760 0x562feb9c1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- 172.21.15.2:0/1846013676 >> 172.21.15.2:0/1846013676 conn(0x562feb9b2cd0 msgr2=0x562feb9bc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:36.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- 172.21.15.2:0/1846013676 shutdown_connections 2024-03-11T18:30:36.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- 172.21.15.2:0/1846013676 wait complete. 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 -- start start 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.004+0000 7fb522a6a740 1 Processor -- start 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 0x562feba23c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb9c8d60 con 0x562feb9b7760 2024-03-11T18:30:36.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feb9c8ed0 con 0x562feb9b8150 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x562feba24190 con 0x562feb9bd2c0 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb522a6a740 10 client.0 _ll_register_callbacks cb 0x7fb510085040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46282/0 (socket says 172.21.15.2:46282) 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 -- 172.21.15.2:0/351332012 learned_addr learned my addr 172.21.15.2:0/351332012 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51ca00640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51de00640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 0x562feba23c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 -- 172.21.15.2:0/351332012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 msgr2=0x562feba23c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 0x562feba23c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 -- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 msgr2=0x562feb9c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9c5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51d400640 1 -- 172.21.15.2:0/351332012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562feb8e74c0 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.005+0000 7fb51de00640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 0x562feba23c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb51d400640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009230 tx=0x7fb50c042070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964a5c917edf9c43 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb4faa00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c051040 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb50c0043d0 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb508011180 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb50c057380 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb522a6a740 10 client.14598 fetch_fsmap learned FSMap version 17 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.006+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562feb8e0510 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb508011180 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fb50c048040 con 0x562feb9b7760 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 10 client.14598 fetch_fsmap finished waiting for FSMap version 17 2024-03-11T18:30:36.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 10 client.14598 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 20 client.14598 populate_metadata read hostname 'smithi002' 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feba29510 0x562feba25930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 4 client.14598 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562feb670270 con 0x562feba29510 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.007+0000 7fb51ca00640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feba29510 0x562feba25930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb51ca00640 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feba29510 0x562feba25930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d169bc55a3f66ab server_cookie=fb8fe3e359391e70 in_seq=0 out_seq=0 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb4faa00640 10 client.14598 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb4faa00640 1 -- 172.21.15.2:0/351332012 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x562feb670270 con 0x562feba29510 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb4faa00640 10 client.14598 handle_command_reply: tid=0 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 client.14598 shutdown 2024-03-11T18:30:36.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 msgr2=0x562feb9c5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009230 tx=0x7fb50c042070 comp rx=0 tx=0).stop 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 shutdown_connections 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x562feba29510 0x562feba25930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x562feb9bd2c0 0x562feba23c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562feb9b8150 0x562feb9c5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb4faa00640 0 client.14598 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 --2- 172.21.15.2:0/351332012 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x562feb9b7760 0x562feb9c5800 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 >> 172.21.15.2:0/351332012 conn(0x562feb9b2cd0 msgr2=0x562feb9b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 shutdown_connections 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.008+0000 7fb522a6a740 1 -- 172.21.15.2:0/351332012 wait complete. 2024-03-11T18:30:36.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb522a6a740 20 client.14598 trim_cache size 0 max 16384 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24575, 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 7.7422925410000003, 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 17, 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 91, 2024-03-11T18:30:36.017 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:30:36.018 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 14.218117948 2024-03-11T18:30:36.018 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:30:36.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 msgr2=0x7fb5000a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:36.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 0x7fb5000a3ef0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb50c02c240 tx=0x7fb50c02c7e0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 msgr2=0x7fb5181a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb51000eb10 tx=0x7fb51000c2d0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 shutdown_connections 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fb5000a1a30 0x7fb5000a3ef0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb518104d90 0x7fb5181a0870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5181043d0 0x7fb518199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 --2- 172.21.15.2:0/763555941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5181031e0 0x7fb5181992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 >> 172.21.15.2:0/763555941 conn(0x7fb5180fe750 msgr2=0x7fb5181076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 shutdown_connections 2024-03-11T18:30:36.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:36.009+0000 7fb51e800640 1 -- 172.21.15.2:0/763555941 wait complete. 2024-03-11T18:30:36.027 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T18:30:36.027 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:36.028 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T18:30:36.489 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:30:36.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:36 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3784916137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:36.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:36 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3784916137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:36.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:36 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3784916137' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:37.573 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:37 smithi165 ceph-mon[36634]: pgmap v325: 129 pgs: 124 active+clean, 5 creating+peering; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-11T18:30:37.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:37 smithi023 ceph-mon[36406]: pgmap v325: 129 pgs: 124 active+clean, 5 creating+peering; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-11T18:30:37.698 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:37 smithi002 ceph-mon[31328]: pgmap v325: 129 pgs: 124 active+clean, 5 creating+peering; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-11T18:30:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.909+0000 7f89e8c00700 1 -- 172.21.15.2:0/3252313267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e4112a60 msgr2=0x7f89e4114e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.909+0000 7f89e8c00700 1 --2- 172.21.15.2:0/3252313267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e4112a60 0x7f89e4114e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009990 tx=0x7f89dc009ca0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 -- 172.21.15.2:0/3252313267 shutdown_connections 2024-03-11T18:30:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 --2- 172.21.15.2:0/3252313267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e4112a60 0x7f89e4114e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 --2- 172.21.15.2:0/3252313267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e40761a0 0x7f89e4076580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 --2- 172.21.15.2:0/3252313267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 0x7f89e4112520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 -- 172.21.15.2:0/3252313267 >> 172.21.15.2:0/3252313267 conn(0x7f89e40ff8f0 msgr2=0x7f89e4101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 -- 172.21.15.2:0/3252313267 shutdown_connections 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.910+0000 7f89e8c00700 1 -- 172.21.15.2:0/3252313267 wait complete. 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 Processor -- start 2024-03-11T18:30:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 -- start start 2024-03-11T18:30:38.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 0x7f89e419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:38.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 0x7f89e419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 0x7f89e41a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e41a3580 con 0x7f89e4112a60 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e41a36f0 con 0x7f89e4076ac0 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.911+0000 7f89e8c00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e41a38b0 con 0x7f89e40761a0 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 0x7f89e41a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 0x7f89e41a2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43510/0 (socket says 172.21.15.2:43510) 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3e00700 1 -- 172.21.15.2:0/1983098638 learned_addr learned my addr 172.21.15.2:0/1983098638 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 0x7f89e419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89daa00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 0x7f89e419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 -- 172.21.15.2:0/1983098638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 msgr2=0x7f89e419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 0x7f89e419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 -- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 msgr2=0x7f89e41a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 0x7f89e41a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 -- 172.21.15.2:0/1983098638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d4009c40 con 0x7f89e40761a0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.912+0000 7f89e3400700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 0x7f89e419e6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89d400efc0 tx=0x7f89d400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.913+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4018070 con 0x7f89e40761a0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.913+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89d400f040 con 0x7f89e40761a0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.913+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89dc0095f0 con 0x7f89e40761a0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.913+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d40042a0 con 0x7f89e40761a0 2024-03-11T18:30:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.913+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e41ab220 con 0x7f89e40761a0 2024-03-11T18:30:38.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.914+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e410dc80 con 0x7f89e40761a0 2024-03-11T18:30:38.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.916+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f89d400ca20 con 0x7f89e40761a0 2024-03-11T18:30:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.917+0000 7f89e0c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 0x7f89c80a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.917+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f89d40cd380 con 0x7f89e40761a0 2024-03-11T18:30:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.917+0000 7f89daa00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 0x7f89c80a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.918+0000 7f89daa00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 0x7f89c80a4000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89e4103350 tx=0x7f89cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:38.921+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f89d4096380 con 0x7f89e40761a0 2024-03-11T18:30:39.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.200+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f89e4068930 con 0x7f89e40761a0 2024-03-11T18:30:39.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.200+0000 7f89e0c00700 1 -- 172.21.15.2:0/1983098638 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f89d4099a40 con 0x7f89e40761a0 2024-03-11T18:30:39.203 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-11T18:30:39.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.203+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 msgr2=0x7f89c80a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:39.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.203+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 0x7f89c80a4000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f89e4103350 tx=0x7f89cc009260 comp rx=0 tx=0).stop 2024-03-11T18:30:39.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.203+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 msgr2=0x7f89e419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.203+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 0x7f89e419e6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89d400efc0 tx=0x7f89d400c3a0 comp rx=0 tx=0).stop 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 shutdown_connections 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89c80a1b40 0x7f89c80a4000 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89e40761a0 0x7f89e419e6a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89e4112a60 0x7f89e41a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 --2- 172.21.15.2:0/1983098638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89e4076ac0 0x7f89e419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 >> 172.21.15.2:0/1983098638 conn(0x7f89e40ff8f0 msgr2=0x7f89e4101ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 shutdown_connections 2024-03-11T18:30:39.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:39.204+0000 7f89e8c00700 1 -- 172.21.15.2:0/1983098638 wait complete. 2024-03-11T18:30:39.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:39 smithi002 ceph-mon[31328]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 296 B/s wr, 4 op/s 2024-03-11T18:30:39.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:39 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1983098638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:30:39.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:39 smithi023 ceph-mon[36406]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 296 B/s wr, 4 op/s 2024-03-11T18:30:39.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:39 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1983098638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:30:39.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:39 smithi165 ceph-mon[36634]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 296 B/s wr, 4 op/s 2024-03-11T18:30:39.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:39 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1983098638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:30:40.117 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T18:30:40.118 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-11T18:30:40.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:40.119 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-03-11T18:30:40.177 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T18:30:40.177 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T18:30:40.177 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-03-11T18:30:40.178 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42073sec preferred_lft 42073sec 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:30:40.179 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-03-11T18:30:40.180 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T18:30:40.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T18:30:40.181 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:30:40.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:40.646 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T18:30:40.730 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T18:30:40.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:40.733 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-03-11T18:30:40.795 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-03-11T18:30:40.795 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T18:30:40.795 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-11T18:30:40.796 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-11T18:30:40.796 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T18:30:40.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-11T18:30:40.797 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:30:41.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:41 smithi002 ceph-mon[31328]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-11T18:30:41.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:41.526 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-03-11T18:30:41.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:41.583 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2024-03-11T18:30:41.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-11T18:30:41.635 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:30:41.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:41 smithi023 ceph-mon[36406]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-11T18:30:41.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:41 smithi165 ceph-mon[36634]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 2 op/s 2024-03-11T18:30:41.782 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-11T18:30:41.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:41.782 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:30:41.782 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:30:41.782 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-11T18:30:41.782 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T18:30:41.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-11T18:30:41.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-11T18:30:41.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-11T18:30:41.783 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:30:42.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:42.074 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:30:42.074 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:30:42.074 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-03-11T18:30:42.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T18:30:42.075 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:30:42.263 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-11T18:30:42.263 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T18:30:42.263 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:42.305 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-11T18:30:42.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T18:30:42.306 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:42.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-03-11T18:30:42.500 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T18:30:42.550 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-03-11T18:30:42.550 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-03-11T18:30:42.550 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/bpf 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-03-11T18:30:42.551 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/71d2fa8f34aca94a9f78e679ad1a359b8f14316c26e77f302502093bbc9a8d8e/merged 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/06981b3b5e5fc44b4e7cae9d8df9691d183438165d7d590a4c80b2ee32fc1ae1/merged 2024-03-11T18:30:42.552 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/0 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/043af2c2d062309020fe4234c125324496afe3d1881daf08ff498f9cfe3a7d53/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/b5832165776023a0a7d9cff104b1b4ec087f71b9d640cb5f01475a7116b01f1d/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/104f2cb35c784a68a88fe3525f3cd0132c2a217d845c23c7f042f515f8c2d041/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/ca9d8131a3537beaaa4d8df1373dd25785c00bb43bb030ae69c39a7fcd3108db/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/1b35c68e553fb282a3d9af623a89e542e4aedb41db709ab34052482f1ebd52e4/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/311754e3e21612d5cb3879e835ca336ccf4d1af6f45cdcd72b2f956fe9b32be6/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/5ddaa24df237b5328ce7777aba751a9ca71ea2324fdce475cc3081d38bc3e57a/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/7f752c5bf3cf5b8363314623c96d5bbe1238b2cc665a5afdd7f62b43637365e9/merged 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:42.553 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:42.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:42.554 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-03-11T18:30:42.594 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-11T18:30:42.594 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-11T18:30:42.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-03-11T18:30:42.721 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/bpf 2024-03-11T18:30:42.767 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T18:30:42.768 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/71d2fa8f34aca94a9f78e679ad1a359b8f14316c26e77f302502093bbc9a8d8e/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/06981b3b5e5fc44b4e7cae9d8df9691d183438165d7d590a4c80b2ee32fc1ae1/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/0 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/043af2c2d062309020fe4234c125324496afe3d1881daf08ff498f9cfe3a7d53/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/b5832165776023a0a7d9cff104b1b4ec087f71b9d640cb5f01475a7116b01f1d/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/104f2cb35c784a68a88fe3525f3cd0132c2a217d845c23c7f042f515f8c2d041/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/ca9d8131a3537beaaa4d8df1373dd25785c00bb43bb030ae69c39a7fcd3108db/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/1b35c68e553fb282a3d9af623a89e542e4aedb41db709ab34052482f1ebd52e4/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/311754e3e21612d5cb3879e835ca336ccf4d1af6f45cdcd72b2f956fe9b32be6/merged 2024-03-11T18:30:42.769 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/5ddaa24df237b5328ce7777aba751a9ca71ea2324fdce475cc3081d38bc3e57a/merged 2024-03-11T18:30:42.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/7f752c5bf3cf5b8363314623c96d5bbe1238b2cc665a5afdd7f62b43637365e9/merged 2024-03-11T18:30:42.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-03-11T18:30:42.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:42.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:42.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:42.771 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-03-11T18:30:42.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:2024-03-11T18:30:42.962+0000 7fe5c223f480 -1 init, newargv = 0x55be2bc60be0 newargc=15 2024-03-11T18:30:42.965 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[76058]: starting ceph client 2024-03-11T18:30:42.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[76058]: starting fuse 2024-03-11T18:30:43.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:43 smithi002 ceph-mon[31328]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 2 op/s 2024-03-11T18:30:43.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:43 smithi165 ceph-mon[36634]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 2 op/s 2024-03-11T18:30:43.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:43 smithi023 ceph-mon[36406]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 2 op/s 2024-03-11T18:30:43.961 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-03-11T18:30:44.066 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-03-11T18:30:44.109 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/bpf 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-11T18:30:44.110 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/71d2fa8f34aca94a9f78e679ad1a359b8f14316c26e77f302502093bbc9a8d8e/merged 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/06981b3b5e5fc44b4e7cae9d8df9691d183438165d7d590a4c80b2ee32fc1ae1/merged 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/0 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/043af2c2d062309020fe4234c125324496afe3d1881daf08ff498f9cfe3a7d53/merged 2024-03-11T18:30:44.111 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/b5832165776023a0a7d9cff104b1b4ec087f71b9d640cb5f01475a7116b01f1d/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/104f2cb35c784a68a88fe3525f3cd0132c2a217d845c23c7f042f515f8c2d041/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/ca9d8131a3537beaaa4d8df1373dd25785c00bb43bb030ae69c39a7fcd3108db/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/1b35c68e553fb282a3d9af623a89e542e4aedb41db709ab34052482f1ebd52e4/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/311754e3e21612d5cb3879e835ca336ccf4d1af6f45cdcd72b2f956fe9b32be6/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/5ddaa24df237b5328ce7777aba751a9ca71ea2324fdce475cc3081d38bc3e57a/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/containers/storage/overlay/7f752c5bf3cf5b8363314623c96d5bbe1238b2cc665a5afdd7f62b43637365e9/merged 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:30:44.112 INFO:teuthology.orchestra.run.smithi002.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:44.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:44.113 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-03-11T18:30:44.205 INFO:teuthology.orchestra.run.smithi002.stdout:118 2024-03-11T18:30:44.206 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> 2024-03-11T18:30:44.206 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-03-11T18:30:44.207 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-03-11T18:30:44.208 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-11T18:30:44.208 DEBUG:teuthology.orchestra.run.smithi002:> 2024-03-11T18:30:44.208 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.admin")) 2024-03-11T18:30:44.208 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-03-11T18:30:44.363 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.admin.76058.asok 2024-03-11T18:30:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-11T18:30:44.380 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76058.asok 2024-03-11T18:30:44.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:44.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76058.asok status 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "admin", 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-03-11T18:30:44.622 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "76058", 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24701, 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 24701 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-03-11T18:30:44.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1850322751 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1850322751 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.24701 172.21.15.2:0/1850322751", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "172.21.15.2:0/1850322751", 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 17, 2024-03-11T18:30:44.624 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 91, 2024-03-11T18:30:44.625 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-03-11T18:30:44.625 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-03-11T18:30:44.625 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-03-11T18:30:44.625 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:30:44.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:44.637 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-03-11T18:30:44.637 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:44.692 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-03-11T18:30:44.692 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:44.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:44.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:44.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:44.793 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-11T18:30:44.883 INFO:teuthology.orchestra.run.smithi002.stdout:# file: . 2024-03-11T18:30:44.884 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-11T18:30:44.884 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:44.885 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:30:44.885 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T18:30:44.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 -- 172.21.15.2:0/1490220555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 msgr2=0x7f4f04108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 --2- 172.21.15.2:0/1490220555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f04108d10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009900 tx=0x7f4ef002bc40 comp rx=0 tx=0).stop 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 -- 172.21.15.2:0/1490220555 shutdown_connections 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 --2- 172.21.15.2:0/1490220555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f0410fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 --2- 172.21.15.2:0/1490220555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f04108d10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 --2- 172.21.15.2:0/1490220555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f04107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 -- 172.21.15.2:0/1490220555 >> 172.21.15.2:0/1490220555 conn(0x7f4f04076640 msgr2=0x7f4f04078a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:45.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.167+0000 7f4f08c00640 1 -- 172.21.15.2:0/1490220555 shutdown_connections 2024-03-11T18:30:45.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.168+0000 7f4f08c00640 1 -- 172.21.15.2:0/1490220555 wait complete. 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 Processor -- start 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 -- start start 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f0419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f041a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041a51f0 con 0x7f4f041076a0 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041a5360 con 0x7f4f04109250 2024-03-11T18:30:45.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.169+0000 7f4f08c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f041a5520 con 0x7f4f04108890 2024-03-11T18:30:45.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43528/0 (socket says 172.21.15.2:43528) 2024-03-11T18:30:45.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 -- 172.21.15.2:0/195905471 learned_addr learned my addr 172.21.15.2:0/195905471 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:45.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03e00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f041a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f02a00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f0419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 -- 172.21.15.2:0/195905471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 msgr2=0x7f4f0419dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f0419dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 -- 172.21.15.2:0/195905471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 msgr2=0x7f4f041a4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f041a4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03400640 1 -- 172.21.15.2:0/195905471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4efc009840 con 0x7f4f041076a0 2024-03-11T18:30:45.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f03e00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f041a4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.170+0000 7f4f02a00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f0419dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4f03400640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400d950 tx=0x7f4ef400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4009c30 con 0x7f4f041076a0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef0009560 con 0x7f4f041076a0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ef400de80 con 0x7f4f041076a0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef400f6b0 con 0x7f4f041076a0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.171+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f041a5b50 con 0x7f4f041076a0 2024-03-11T18:30:45.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.172+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0410e9c0 con 0x7f4f041076a0 2024-03-11T18:30:45.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.174+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f4ef4010460 con 0x7f4f041076a0 2024-03-11T18:30:45.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.176+0000 7f4efbe00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 0x7f4ee80a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.176+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4ef40cd810 con 0x7f4f041076a0 2024-03-11T18:30:45.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.176+0000 7f4f02a00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 0x7f4ee80a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.177+0000 7f4f02a00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 0x7f4ee80a3ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009230 tx=0x7f4ef0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:45.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.181+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ef4096780 con 0x7f4f041076a0 2024-03-11T18:30:45.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.390+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f041130b0 con 0x7f4f041076a0 2024-03-11T18:30:45.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.394+0000 7f4efbe00640 1 -- 172.21.15.2:0/195905471 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f4ef4099e40 con 0x7f4f041076a0 2024-03-11T18:30:45.397 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:45.397 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 msgr2=0x7f4ee80a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 0x7f4ee80a3ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009230 tx=0x7f4ef0005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 msgr2=0x7f4f0419d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400d950 tx=0x7f4ef400dd50 comp rx=0 tx=0).stop 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 shutdown_connections 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4ee80a1a20 0x7f4ee80a3ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f04109250 0x7f4f041a4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f04108890 0x7f4f0419dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 --2- 172.21.15.2:0/195905471 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f041076a0 0x7f4f0419d7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 >> 172.21.15.2:0/195905471 conn(0x7f4f04076640 msgr2=0x7f4f0410bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 shutdown_connections 2024-03-11T18:30:45.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.397+0000 7f4f08c00640 1 -- 172.21.15.2:0/195905471 wait complete. 2024-03-11T18:30:45.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:30:45.420 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:30:45.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 -- 172.21.15.2:0/3536410615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4380101a20 msgr2=0x7f4380108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 --2- 172.21.15.2:0/3536410615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4380101a20 0x7f4380108340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4378009900 tx=0x7f437802bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 -- 172.21.15.2:0/3536410615 shutdown_connections 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 --2- 172.21.15.2:0/3536410615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4380101a20 0x7f4380108340 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 --2- 172.21.15.2:0/3536410615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 0x7f43801014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 --2- 172.21.15.2:0/3536410615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43800ffe80 0x7f43801002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 -- 172.21.15.2:0/3536410615 >> 172.21.15.2:0/3536410615 conn(0x7f43800fb460 msgr2=0x7f43800fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:45.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 -- 172.21.15.2:0/3536410615 shutdown_connections 2024-03-11T18:30:45.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.649+0000 7f4385200640 1 -- 172.21.15.2:0/3536410615 wait complete. 2024-03-11T18:30:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 Processor -- start 2024-03-11T18:30:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 -- start start 2024-03-11T18:30:45.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 0x7f4380199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4380101a20 0x7f43801a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801a0d30 con 0x7f4380101060 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801a0ea0 con 0x7f43800ffe80 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.650+0000 7f4385200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43801a1060 con 0x7f4380101a20 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:53318/0 (socket says 172.21.15.2:53318) 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 -- 172.21.15.2:0/1250986743 learned_addr learned my addr 172.21.15.2:0/1250986743 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437ea00640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 0x7f4380199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437fe00640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4380101a20 0x7f43801a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 -- 172.21.15.2:0/1250986743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4380101a20 msgr2=0x7f43801a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4380101a20 0x7f43801a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 -- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 msgr2=0x7f4380199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 0x7f4380199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 -- 172.21.15.2:0/1250986743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4378009560 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.651+0000 7f437f400640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f437000e9c0 tx=0x7f437000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.652+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43700096e0 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.652+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f437000cd80 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.652+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370010430 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.652+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43801a1340 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.652+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43801a1800 con 0x7f43800ffe80 2024-03-11T18:30:45.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.653+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438010b8e0 con 0x7f43800ffe80 2024-03-11T18:30:45.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:45 smithi023 ceph-mon[36406]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 90 B/s wr, 1 op/s 2024-03-11T18:30:45.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:45 smithi165 ceph-mon[36634]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 90 B/s wr, 1 op/s 2024-03-11T18:30:45.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.655+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f4370003680 con 0x7f43800ffe80 2024-03-11T18:30:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.656+0000 7f4377e00640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 0x7f43600a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.656+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4370014070 con 0x7f43800ffe80 2024-03-11T18:30:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.657+0000 7f437ea00640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 0x7f43600a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:45.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.657+0000 7f437ea00640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 0x7f43600a3f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f438019a750 tx=0x7f4368009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:45.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.661+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43700969f0 con 0x7f43800ffe80 2024-03-11T18:30:45.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:45 smithi002 ceph-mon[31328]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 90 B/s wr, 1 op/s 2024-03-11T18:30:45.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.860+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43801a1d80 con 0x7f43800ffe80 2024-03-11T18:30:45.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.862+0000 7f4377e00640 1 -- 172.21.15.2:0/1250986743 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f437009a0b0 con 0x7f43800ffe80 2024-03-11T18:30:45.863 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 msgr2=0x7f43600a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 0x7f43600a3f70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f438019a750 tx=0x7f4368009500 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 msgr2=0x7f4380199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f437000e9c0 tx=0x7f437000ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 shutdown_connections 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f43600a1ab0 0x7f43600a3f70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4380101a20 0x7f43801a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4380101060 0x7f4380199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 --2- 172.21.15.2:0/1250986743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43800ffe80 0x7f4380199240 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 >> 172.21.15.2:0/1250986743 conn(0x7f43800fb460 msgr2=0x7f43800fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 shutdown_connections 2024-03-11T18:30:45.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:45.864+0000 7f4385200640 1 -- 172.21.15.2:0/1250986743 wait complete. 2024-03-11T18:30:45.885 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:30:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 -- 172.21.15.2:0/2827900733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 msgr2=0x7fa6fc107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 --2- 172.21.15.2:0/2827900733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc107be0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009900 tx=0x7fa6ec02bc50 comp rx=0 tx=0).stop 2024-03-11T18:30:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 -- 172.21.15.2:0/2827900733 shutdown_connections 2024-03-11T18:30:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 --2- 172.21.15.2:0/2827900733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 --2- 172.21.15.2:0/2827900733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 0x7fa6fc108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 --2- 172.21.15.2:0/2827900733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc107be0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 -- 172.21.15.2:0/2827900733 >> 172.21.15.2:0/2827900733 conn(0x7fa6fc076770 msgr2=0x7fa6fc078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.111+0000 7fa700e00640 1 -- 172.21.15.2:0/2827900733 shutdown_connections 2024-03-11T18:30:46.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.112+0000 7fa700e00640 1 -- 172.21.15.2:0/2827900733 wait complete. 2024-03-11T18:30:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 Processor -- start 2024-03-11T18:30:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 -- start start 2024-03-11T18:30:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 0x7fa6fc19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc1a5360 con 0x7fa6fc109370 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc1a54d0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa700e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc1a5690 con 0x7fa6fc1089b0 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.113+0000 7fa6fa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6fa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43582/0 (socket says 172.21.15.2:43582) 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 0x7fa6fc19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6fa800640 1 -- 172.21.15.2:0/964634749 learned_addr learned my addr 172.21.15.2:0/964634749 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 -- 172.21.15.2:0/964634749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 msgr2=0x7fa6fc19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 0x7fa6fc19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 -- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 msgr2=0x7fa6fc1a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 -- 172.21.15.2:0/964634749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec009560 con 0x7fa6fc1077c0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6fa800640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.114+0000 7fa6f9e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc19d8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec036040 tx=0x7fa6ec005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.115+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0043d0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.115+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6fc1a5910 con 0x7fa6fc1077c0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.115+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6ec0052d0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.115+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec003cd0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.115+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc1a5df0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.116+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6fc10eb60 con 0x7fa6fc1077c0 2024-03-11T18:30:46.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.118+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa6ec005440 con 0x7fa6fc1077c0 2024-03-11T18:30:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.119+0000 7fa6f2a00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 0x7fa6e00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.119+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa6ec0effe0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.120+0000 7fa6f9400640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 0x7fa6e00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.120+0000 7fa6f9400640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 0x7fa6e00a3ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc19ed80 tx=0x7fa6e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:46.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.122+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa6ec0b8fb0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.323+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6fc02cdd0 con 0x7fa6fc1077c0 2024-03-11T18:30:46.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.324+0000 7fa6f2a00640 1 -- 172.21.15.2:0/964634749 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa6ec043020 con 0x7fa6fc1077c0 2024-03-11T18:30:46.327 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 msgr2=0x7fa6e00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 0x7fa6e00a3ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc19ed80 tx=0x7fa6e8005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 msgr2=0x7fa6fc19d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc19d8c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec036040 tx=0x7fa6ec005e80 comp rx=0 tx=0).stop 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 shutdown_connections 2024-03-11T18:30:46.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa6e00a1a30 0x7fa6e00a3ef0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6fc109370 0x7fa6fc1a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6fc1089b0 0x7fa6fc19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 --2- 172.21.15.2:0/964634749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6fc1077c0 0x7fa6fc19d8c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 >> 172.21.15.2:0/964634749 conn(0x7fa6fc076770 msgr2=0x7fa6fc10bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 shutdown_connections 2024-03-11T18:30:46.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.327+0000 7fa700e00640 1 -- 172.21.15.2:0/964634749 wait complete. 2024-03-11T18:30:46.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 -- 172.21.15.2:0/1849305011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101089b0 msgr2=0x7f7110108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 --2- 172.21.15.2:0/1849305011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101089b0 0x7f7110108e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009900 tx=0x7f70fc02bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 -- 172.21.15.2:0/1849305011 shutdown_connections 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 --2- 172.21.15.2:0/1849305011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 0x7f711010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 --2- 172.21.15.2:0/1849305011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101089b0 0x7f7110108e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 --2- 172.21.15.2:0/1849305011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101077c0 0x7f7110107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 -- 172.21.15.2:0/1849305011 >> 172.21.15.2:0/1849305011 conn(0x7f7110076770 msgr2=0x7f7110078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:46.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 -- 172.21.15.2:0/1849305011 shutdown_connections 2024-03-11T18:30:46.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.567+0000 7f7118400640 1 -- 172.21.15.2:0/1849305011 wait complete. 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.568+0000 7f7118400640 1 Processor -- start 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.568+0000 7f7118400640 1 -- start start 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 0x7f711019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 0x7f71101a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71101a5360 con 0x7f71101077c0 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71101a54d0 con 0x7f71101089b0 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7118400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71101a5690 con 0x7f7110109370 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43608/0 (socket says 172.21.15.2:43608) 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 0x7f711019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115a00640 1 -- 172.21.15.2:0/777144107 learned_addr learned my addr 172.21.15.2:0/777144107 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:46.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7116400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 0x7f71101a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115000640 1 -- 172.21.15.2:0/777144107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 msgr2=0x7f71101a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115000640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 0x7f71101a4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.569+0000 7f7115000640 1 -- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 msgr2=0x7f711019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7115000640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7115000640 1 -- 172.21.15.2:0/777144107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc009560 con 0x7f71101089b0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7115a00640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7115000640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 0x7f711019ddb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f70fc000c00 tx=0x7f70fc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc004aa0 con 0x7f71101089b0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71101a5910 con 0x7f71101089b0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70fc0052d0 con 0x7f71101089b0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc003e80 con 0x7f71101089b0 2024-03-11T18:30:46.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.570+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71101a5e40 con 0x7f71101089b0 2024-03-11T18:30:46.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.571+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711010eb60 con 0x7f71101089b0 2024-03-11T18:30:46.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.573+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f70fc0035f0 con 0x7f71101089b0 2024-03-11T18:30:46.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.575+0000 7f710e000640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 0x7f70f80a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:46.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.575+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f70fc0efc50 con 0x7f71101089b0 2024-03-11T18:30:46.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.575+0000 7f7115a00640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 0x7f70f80a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:46.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.576+0000 7f7115a00640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 0x7f70f80a3f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f71040058c0 tx=0x7f7104005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:46.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.578+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f70fc0b8bc0 con 0x7f71101089b0 2024-03-11T18:30:46.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/195905471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:46.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1250986743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/964634749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/195905471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:46.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1250986743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/964634749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/195905471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:46.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1250986743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/964634749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:46.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.781+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71101133d0 con 0x7f71101089b0 2024-03-11T18:30:46.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.783+0000 7f710e000640 1 -- 172.21.15.2:0/777144107 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70fc043020 con 0x7f71101089b0 2024-03-11T18:30:46.784 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 msgr2=0x7f70f80a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 0x7f70f80a3f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f71040058c0 tx=0x7f7104005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 msgr2=0x7f711019ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 0x7f711019ddb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f70fc000c00 tx=0x7f70fc004270 comp rx=0 tx=0).stop 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 shutdown_connections 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f70f80a1ab0 0x7f70f80a3f70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7110109370 0x7f71101a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71101089b0 0x7f711019ddb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 --2- 172.21.15.2:0/777144107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71101077c0 0x7f711019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:46.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.784+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 >> 172.21.15.2:0/777144107 conn(0x7f7110076770 msgr2=0x7f711010bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:46.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.785+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 shutdown_connections 2024-03-11T18:30:46.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:46.785+0000 7f7118400640 1 -- 172.21.15.2:0/777144107 wait complete. 2024-03-11T18:30:46.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:30:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 -- 172.21.15.2:0/1692144815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 msgr2=0x7f708410fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 --2- 172.21.15.2:0/1692144815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f708410fa20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f707c009ca0 tx=0x7f707c02c100 comp rx=0 tx=0).stop 2024-03-11T18:30:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 -- 172.21.15.2:0/1692144815 shutdown_connections 2024-03-11T18:30:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 --2- 172.21.15.2:0/1692144815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f708410fa20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 --2- 172.21.15.2:0/1692144815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841087d0 0x7f7084108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 --2- 172.21.15.2:0/1692144815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841075e0 0x7f7084107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 -- 172.21.15.2:0/1692144815 >> 172.21.15.2:0/1692144815 conn(0x7f7084076570 msgr2=0x7f70840789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:47.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 -- 172.21.15.2:0/1692144815 shutdown_connections 2024-03-11T18:30:47.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.025+0000 7f7089e00640 1 -- 172.21.15.2:0/1692144815 wait complete. 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.026+0000 7f7089e00640 1 Processor -- start 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.026+0000 7f7089e00640 1 -- start start 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 0x7f708419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f70841a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841a5150 con 0x7f7084109190 2024-03-11T18:30:47.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841a52c0 con 0x7f70841075e0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70841a5480 con 0x7f70841087d0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7083e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7089400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f70841a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7088a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 0x7f708419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7083e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41680/0 (socket says 172.21.15.2:41680) 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.027+0000 7f7083e00640 1 -- 172.21.15.2:0/1203139284 learned_addr learned my addr 172.21.15.2:0/1203139284 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 -- 172.21.15.2:0/1203139284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 msgr2=0x7f708419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 0x7f708419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 -- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 msgr2=0x7f70841a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f70841a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 -- 172.21.15.2:0/1203139284 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078009490 con 0x7f70841087d0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7088a00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 0x7f708419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7083e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:47.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070009700 con 0x7f70841087d0 2024-03-11T18:30:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c009900 con 0x7f70841087d0 2024-03-11T18:30:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f707000cd80 con 0x7f70841087d0 2024-03-11T18:30:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.028+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070018920 con 0x7f70841087d0 2024-03-11T18:30:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.029+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70841a5b30 con 0x7f70841087d0 2024-03-11T18:30:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.030+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708410e8a0 con 0x7f70841087d0 2024-03-11T18:30:47.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.032+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f7070018a80 con 0x7f70841087d0 2024-03-11T18:30:47.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.035+0000 7f7081600640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 0x7f706c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.035+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7070014070 con 0x7f70841087d0 2024-03-11T18:30:47.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.035+0000 7f7088a00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 0x7f706c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.035+0000 7f7088a00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 0x7f706c0a3ef0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7084108630 tx=0x7f7078005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:47.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.035+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7070096ed0 con 0x7f70841087d0 2024-03-11T18:30:47.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.239+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7084112ff0 con 0x7f70841087d0 2024-03-11T18:30:47.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.240+0000 7f7081600640 1 -- 172.21.15.2:0/1203139284 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f707009a590 con 0x7f70841087d0 2024-03-11T18:30:47.242 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:30:47.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 msgr2=0x7f706c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 0x7f706c0a3ef0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7084108630 tx=0x7f7078005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 msgr2=0x7f708419dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f707000e9f0 tx=0x7f707000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 shutdown_connections 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f706c0a1a30 0x7f706c0a3ef0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7084109190 0x7f70841a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f70841087d0 0x7f708419dba0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 --2- 172.21.15.2:0/1203139284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70841075e0 0x7f708419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.242+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 >> 172.21.15.2:0/1203139284 conn(0x7f7084076570 msgr2=0x7f708410ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:47.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.243+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 shutdown_connections 2024-03-11T18:30:47.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.243+0000 7f7089e00640 1 -- 172.21.15.2:0/1203139284 wait complete. 2024-03-11T18:30:47.265 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 -- 172.21.15.2:0/1018018261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c106cb0 msgr2=0x7f252c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 --2- 172.21.15.2:0/1018018261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c106cb0 0x7f252c109100 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2524009ca0 tx=0x7f252402c100 comp rx=0 tx=0).stop 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 -- 172.21.15.2:0/1018018261 shutdown_connections 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 --2- 172.21.15.2:0/1018018261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c106cb0 0x7f252c109100 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 --2- 172.21.15.2:0/1018018261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c104380 0x7f252c106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 --2- 172.21.15.2:0/1018018261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c101a50 0x7f252c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 -- 172.21.15.2:0/1018018261 >> 172.21.15.2:0/1018018261 conn(0x7f252c0fb460 msgr2=0x7f252c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:47.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.494+0000 7f2532000640 1 -- 172.21.15.2:0/1018018261 shutdown_connections 2024-03-11T18:30:47.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.495+0000 7f2532000640 1 -- 172.21.15.2:0/1018018261 wait complete. 2024-03-11T18:30:47.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 Processor -- start 2024-03-11T18:30:47.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 -- start start 2024-03-11T18:30:47.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 0x7f252c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c106cb0 0x7f252c1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a0d30 con 0x7f252c101a50 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a0ea0 con 0x7f252c104380 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.496+0000 7f2532000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a1060 con 0x7f252c106cb0 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f2530c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 0x7f252c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51552/0 (socket says 172.21.15.2:51552) 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f2530c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 0x7f252c1992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48110/0 (socket says 172.21.15.2:48110) 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 -- 172.21.15.2:0/1127123799 learned_addr learned my addr 172.21.15.2:0/1127123799 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f2531600640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c106cb0 0x7f252c1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 -- 172.21.15.2:0/1127123799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c106cb0 msgr2=0x7f252c1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c106cb0 0x7f252c1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 -- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 msgr2=0x7f252c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 0x7f252c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 -- 172.21.15.2:0/1127123799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524009900 con 0x7f252c104380 2024-03-11T18:30:47.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.497+0000 7f252be00640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f251c00e9f0 tx=0x7f251c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:47.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.498+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c0096c0 con 0x7f252c104380 2024-03-11T18:30:47.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.498+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f251c005500 con 0x7f252c104380 2024-03-11T18:30:47.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.498+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c1a1340 con 0x7f252c104380 2024-03-11T18:30:47.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.498+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c010430 con 0x7f252c104380 2024-03-11T18:30:47.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.498+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c1a1800 con 0x7f252c104380 2024-03-11T18:30:47.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.499+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c1087a0 con 0x7f252c104380 2024-03-11T18:30:47.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.502+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f251c005020 con 0x7f252c104380 2024-03-11T18:30:47.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.502+0000 7f2529600640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 0x7f25100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:47.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.502+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f251c014070 con 0x7f252c104380 2024-03-11T18:30:47.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.502+0000 7f2530c00640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 0x7f25100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:47.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.503+0000 7f2530c00640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 0x7f25100a3ef0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2514009bd0 tx=0x7f2514005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:47.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.507+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f251c096940 con 0x7f252c104380 2024-03-11T18:30:47.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:47 smithi165 ceph-mon[36634]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:47.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:47 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/777144107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:47 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1203139284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:47 smithi023 ceph-mon[36406]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:47.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:47 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/777144107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:47 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1203139284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.721+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f252c10c9e0 con 0x7f252c104380 2024-03-11T18:30:47.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.722+0000 7f2529600640 1 -- 172.21.15.2:0/1127123799 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+581 (secure 0 0 0) 0x7f251c09a000 con 0x7f252c104380 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1969024355 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3822870648 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/247107605 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3920276264 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3359009707 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3303291273 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:47.724 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:47.725 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 msgr2=0x7f25100a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 0x7f25100a3ef0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2514009bd0 tx=0x7f2514005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 msgr2=0x7f252c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f251c00e9f0 tx=0x7f251c00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 shutdown_connections 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f25100a1a30 0x7f25100a3ef0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f252c106cb0 0x7f252c1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f252c104380 0x7f252c1997f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 --2- 172.21.15.2:0/1127123799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f252c101a50 0x7f252c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:47.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 >> 172.21.15.2:0/1127123799 conn(0x7f252c0fb460 msgr2=0x7f252c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 shutdown_connections 2024-03-11T18:30:47.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:47.725+0000 7f2532000640 1 -- 172.21.15.2:0/1127123799 wait complete. 2024-03-11T18:30:47.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:47 smithi002 ceph-mon[31328]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:47.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:47 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/777144107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:47 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1203139284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:47.746 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:30:47.746 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T18:30:47.803 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 -- 172.21.15.2:0/1681427007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 msgr2=0x7f0c6c10b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 --2- 172.21.15.2:0/1681427007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c10b800 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c64009ca0 tx=0x7f0c6402c100 comp rx=0 tx=0).stop 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 -- 172.21.15.2:0/1681427007 shutdown_connections 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 --2- 172.21.15.2:0/1681427007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c10b800 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 --2- 172.21.15.2:0/1681427007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1045b0 0x7f0c6c104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 --2- 172.21.15.2:0/1681427007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1033c0 0x7f0c6c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 -- 172.21.15.2:0/1681427007 >> 172.21.15.2:0/1681427007 conn(0x7f0c6c0fe970 msgr2=0x7f0c6c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 -- 172.21.15.2:0/1681427007 shutdown_connections 2024-03-11T18:30:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.029+0000 7f0c71800640 1 -- 172.21.15.2:0/1681427007 wait complete. 2024-03-11T18:30:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 Processor -- start 2024-03-11T18:30:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 -- start start 2024-03-11T18:30:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1033c0 0x7f0c6c1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1045b0 0x7f0c6c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c1a0ee0 con 0x7f0c6c104f70 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c1a1050 con 0x7f0c6c1033c0 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.030+0000 7f0c71800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c1a1210 con 0x7f0c6c1045b0 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1033c0 0x7f0c6c1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48120/0 (socket says 172.21.15.2:48120) 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 -- 172.21.15.2:0/2372646078 learned_addr learned my addr 172.21.15.2:0/2372646078 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6a000640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1045b0 0x7f0c6c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 -- 172.21.15.2:0/2372646078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1045b0 msgr2=0x7f0c6c199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1045b0 0x7f0c6c199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 -- 172.21.15.2:0/2372646078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1033c0 msgr2=0x7f0c6c1993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1033c0 0x7f0c6c1993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 -- 172.21.15.2:0/2372646078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64009900 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.031+0000 7f0c6b400640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c640 tx=0x7f0c64032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.032+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64034640 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.032+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c64003e50 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.032+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64004840 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.032+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c1a1490 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.032+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c1a18f0 con 0x7f0c6c104f70 2024-03-11T18:30:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.033+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c10a680 con 0x7f0c6c104f70 2024-03-11T18:30:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.035+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f0c6400a040 con 0x7f0c6c104f70 2024-03-11T18:30:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.037+0000 7f0c63400640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 0x7f0c500a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.037+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0c640effd0 con 0x7f0c6c104f70 2024-03-11T18:30:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.038+0000 7f0c6aa00640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 0x7f0c500a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.038+0000 7f0c6aa00640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 0x7f0c500a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c104410 tx=0x7f0c5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.042+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0c640b8f40 con 0x7f0c6c104f70 2024-03-11T18:30:48.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.245+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c6c02cea0 con 0x7f0c6c104f70 2024-03-11T18:30:48.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.249+0000 7f0c63400640 1 -- 172.21.15.2:0/2372646078 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f0c640bc600 con 0x7f0c6c104f70 2024-03-11T18:30:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:48.251 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 msgr2=0x7f0c500a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 0x7f0c500a3f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c104410 tx=0x7f0c5c005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 msgr2=0x7f0c6c1a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0c6402c640 tx=0x7f0c64032fc0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 shutdown_connections 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0c500a1ab0 0x7f0c500a3f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c6c104f70 0x7f0c6c1a09a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c6c1045b0 0x7f0c6c199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 --2- 172.21.15.2:0/2372646078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c6c1033c0 0x7f0c6c1993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 >> 172.21.15.2:0/2372646078 conn(0x7f0c6c0fe970 msgr2=0x7f0c6c107850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:48.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 shutdown_connections 2024-03-11T18:30:48.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.252+0000 7f0c71800640 1 -- 172.21.15.2:0/2372646078 wait complete. 2024-03-11T18:30:48.274 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:30:48.275 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:30:48.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 -- 172.21.15.2:0/4045081634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1033b0 msgr2=0x7fe54c1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 --2- 172.21.15.2:0/4045081634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1033b0 0x7fe54c1037d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe544009ca0 tx=0x7fe54402c100 comp rx=0 tx=0).stop 2024-03-11T18:30:48.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 -- 172.21.15.2:0/4045081634 shutdown_connections 2024-03-11T18:30:48.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 --2- 172.21.15.2:0/4045081634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c104f60 0x7fe54c10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 --2- 172.21.15.2:0/4045081634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c1045a0 0x7fe54c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 --2- 172.21.15.2:0/4045081634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1033b0 0x7fe54c1037d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 -- 172.21.15.2:0/4045081634 >> 172.21.15.2:0/4045081634 conn(0x7fe54c0fe940 msgr2=0x7fe54c100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.509+0000 7fe553000640 1 -- 172.21.15.2:0/4045081634 shutdown_connections 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.510+0000 7fe553000640 1 -- 172.21.15.2:0/4045081634 wait complete. 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 Processor -- start 2024-03-11T18:30:48.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 -- start start 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1045a0 0x7fe54c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c19ec10 0x7fe54c1a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1a1540 con 0x7fe54c1045a0 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1a16b0 con 0x7fe54c19ec10 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.511+0000 7fe553000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1a1870 con 0x7fe54c104f60 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1045a0 0x7fe54c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41734/0 (socket says 172.21.15.2:41734) 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe551000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c19ec10 0x7fe54c1a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 -- 172.21.15.2:0/2355715490 learned_addr learned my addr 172.21.15.2:0/2355715490 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 -- 172.21.15.2:0/2355715490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c19ec10 msgr2=0x7fe54c1a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c19ec10 0x7fe54c1a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 -- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1045a0 msgr2=0x7fe54c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1045a0 0x7fe54c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.512+0000 7fe54b400640 1 -- 172.21.15.2:0/2355715490 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544009900 con 0x7fe54c104f60 2024-03-11T18:30:48.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe54b400640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e9f0 tx=0x7fe53400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534009720 con 0x7fe54c104f60 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe53400cd80 con 0x7fe54c104f60 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c1a1b50 con 0x7fe54c104f60 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534010690 con 0x7fe54c104f60 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.513+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c1a2060 con 0x7fe54c104f60 2024-03-11T18:30:48.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.514+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c10a740 con 0x7fe54c104f60 2024-03-11T18:30:48.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.517+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fe5340046c0 con 0x7fe54c104f60 2024-03-11T18:30:48.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.518+0000 7fe548c00640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 0x7fe5300a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.518+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe534014070 con 0x7fe54c104f60 2024-03-11T18:30:48.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.518+0000 7fe54be00640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 0x7fe5300a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.519+0000 7fe54be00640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 0x7fe5300a3f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe544032600 tx=0x7fe544000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.522+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe5340968f0 con 0x7fe54c104f60 2024-03-11T18:30:48.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:48 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1127123799' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:48.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:48 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2372646078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:48.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:48 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1127123799' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:48.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:48 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2372646078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.724+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe54c02ce40 con 0x7fe54c104f60 2024-03-11T18:30:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.725+0000 7fe548c00640 1 -- 172.21.15.2:0/2355715490 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe534099fb0 con 0x7fe54c104f60 2024-03-11T18:30:48.726 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:30:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.727+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 msgr2=0x7fe5300a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.727+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 0x7fe5300a3f70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe544032600 tx=0x7fe544000f40 comp rx=0 tx=0).stop 2024-03-11T18:30:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 msgr2=0x7fe54c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e9f0 tx=0x7fe53400edf0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 shutdown_connections 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe5300a1ab0 0x7fe5300a3f70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe54c19ec10 0x7fe54c1a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe54c104f60 0x7fe54c199bb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 --2- 172.21.15.2:0/2355715490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54c1045a0 0x7fe54c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 >> 172.21.15.2:0/2355715490 conn(0x7fe54c0fe940 msgr2=0x7fe54c107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 shutdown_connections 2024-03-11T18:30:48.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.728+0000 7fe553000640 1 -- 172.21.15.2:0/2355715490 wait complete. 2024-03-11T18:30:48.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:48 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1127123799' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:48.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:48 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2372646078' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:48.748 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 -- 172.21.15.2:0/1634234005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430101a50 msgr2=0x7fd430103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 --2- 172.21.15.2:0/1634234005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430101a50 0x7fd430103e40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd418009900 tx=0x7fd41802bc50 comp rx=0 tx=0).stop 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 -- 172.21.15.2:0/1634234005 shutdown_connections 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 --2- 172.21.15.2:0/1634234005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430106cb0 0x7fd430109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 --2- 172.21.15.2:0/1634234005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430104380 0x7fd430106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 --2- 172.21.15.2:0/1634234005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430101a50 0x7fd430103e40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 -- 172.21.15.2:0/1634234005 >> 172.21.15.2:0/1634234005 conn(0x7fd4300fb460 msgr2=0x7fd4300fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 -- 172.21.15.2:0/1634234005 shutdown_connections 2024-03-11T18:30:48.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.975+0000 7fd436200640 1 -- 172.21.15.2:0/1634234005 wait complete. 2024-03-11T18:30:48.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.976+0000 7fd436200640 1 Processor -- start 2024-03-11T18:30:48.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.976+0000 7fd436200640 1 -- start start 2024-03-11T18:30:48.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430101a50 0x7fd430197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430104380 0x7fd4301975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43019eb50 con 0x7fd430106cb0 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43019ecc0 con 0x7fd430101a50 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd436200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43019ee80 con 0x7fd430104380 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48160/0 (socket says 172.21.15.2:48160) 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 -- 172.21.15.2:0/1439101417 learned_addr learned my addr 172.21.15.2:0/1439101417 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd427e00640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430104380 0x7fd4301975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd434e00640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430101a50 0x7fd430197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 -- 172.21.15.2:0/1439101417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430104380 msgr2=0x7fd4301975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430104380 0x7fd4301975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 -- 172.21.15.2:0/1439101417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430101a50 msgr2=0x7fd430197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:48.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430101a50 0x7fd430197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.977+0000 7fd435800640 1 -- 172.21.15.2:0/1439101417 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd418009560 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd435800640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd42800eaa0 tx=0x7fd42800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42800cdb0 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd428005020 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43019f160 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4280116b0 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.978+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43019f670 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.979+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4300fcf50 con 0x7fd430106cb0 2024-03-11T18:30:48.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.981+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd428005520 con 0x7fd430106cb0 2024-03-11T18:30:48.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.983+0000 7fd42e000640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 0x7fd4100a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:48.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.983+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd4280cdb50 con 0x7fd430106cb0 2024-03-11T18:30:48.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.983+0000 7fd434e00640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 0x7fd4100a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:48.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.984+0000 7fd434e00640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 0x7fd4100a3f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd418009230 tx=0x7fd418005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:48.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:48.986+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd428096bf0 con 0x7fd430106cb0 2024-03-11T18:30:49.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.185+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd43010c9c0 con 0x7fd430106cb0 2024-03-11T18:30:49.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.186+0000 7fd42e000640 1 -- 172.21.15.2:0/1439101417 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd42801f020 con 0x7fd430106cb0 2024-03-11T18:30:49.187 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 msgr2=0x7fd4100a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 0x7fd4100a3f70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd418009230 tx=0x7fd418005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 msgr2=0x7fd43019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd42800eaa0 tx=0x7fd42800ef60 comp rx=0 tx=0).stop 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 shutdown_connections 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd4100a1ab0 0x7fd4100a3f70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd430106cb0 0x7fd43019e610 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd430104380 0x7fd4301975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 --2- 172.21.15.2:0/1439101417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd430101a50 0x7fd430197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.188+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 >> 172.21.15.2:0/1439101417 conn(0x7fd4300fb460 msgr2=0x7fd430102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.189+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 shutdown_connections 2024-03-11T18:30:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.189+0000 7fd436200640 1 -- 172.21.15.2:0/1439101417 wait complete. 2024-03-11T18:30:49.208 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:30:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 -- 172.21.15.2:0/3133954494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681087d0 msgr2=0x7f3e68108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 --2- 172.21.15.2:0/3133954494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681087d0 0x7f3e68108c50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009ca0 tx=0x7f3e5402c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 -- 172.21.15.2:0/3133954494 shutdown_connections 2024-03-11T18:30:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 --2- 172.21.15.2:0/3133954494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e68109190 0x7f3e6810fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 --2- 172.21.15.2:0/3133954494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681087d0 0x7f3e68108c50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 --2- 172.21.15.2:0/3133954494 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e681075e0 0x7f3e68107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 -- 172.21.15.2:0/3133954494 >> 172.21.15.2:0/3133954494 conn(0x7f3e68076570 msgr2=0x7f3e680789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.429+0000 7f3e6e200640 1 -- 172.21.15.2:0/3133954494 shutdown_connections 2024-03-11T18:30:49.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.430+0000 7f3e6e200640 1 -- 172.21.15.2:0/3133954494 wait complete. 2024-03-11T18:30:49.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.430+0000 7f3e6e200640 1 Processor -- start 2024-03-11T18:30:49.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 -- start start 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681075e0 0x7f3e6819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 0x7f3e6819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 0x7f3e681a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681a5140 con 0x7f3e68109190 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681a52b0 con 0x7f3e681075e0 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6e200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681a5470 con 0x7f3e681087d0 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e67e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 0x7f3e6819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e67e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 0x7f3e6819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41770/0 (socket says 172.21.15.2:41770) 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 0x7f3e681a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e67e00640 1 -- 172.21.15.2:0/2493273705 learned_addr learned my addr 172.21.15.2:0/2493273705 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.431+0000 7f3e6ce00640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681075e0 0x7f3e6819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 -- 172.21.15.2:0/2493273705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 msgr2=0x7f3e6819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 0x7f3e6819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 -- 172.21.15.2:0/2493273705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681075e0 msgr2=0x7f3e6819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681075e0 0x7f3e6819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 -- 172.21.15.2:0/2493273705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54009900 con 0x7f3e68109190 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6d800640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 0x7f3e681a4c00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d680 tx=0x7f3e6000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60005220 con 0x7f3e68109190 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e681a5750 con 0x7f3e68109190 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e60009d00 con 0x7f3e68109190 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60004ea0 con 0x7f3e68109190 2024-03-11T18:30:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.432+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e681a5c30 con 0x7f3e68109190 2024-03-11T18:30:49.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.434+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6810e900 con 0x7f3e68109190 2024-03-11T18:30:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.436+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3e60009820 con 0x7f3e68109190 2024-03-11T18:30:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.437+0000 7f3e65600640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 0x7f3e4c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.437+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3e600cd2e0 con 0x7f3e68109190 2024-03-11T18:30:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.437+0000 7f3e6ce00640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 0x7f3e4c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.438+0000 7f3e6ce00640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 0x7f3e4c0a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3e68108630 tx=0x7f3e58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.440+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e60096360 con 0x7f3e68109190 2024-03-11T18:30:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:49 smithi002 ceph-mon[31328]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:49 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2355715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:49 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1439101417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.641+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e68112ff0 con 0x7f3e68109190 2024-03-11T18:30:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.642+0000 7f3e65600640 1 -- 172.21.15.2:0/2493273705 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3e60017020 con 0x7f3e68109190 2024-03-11T18:30:49.643 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:30:49.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.644+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 msgr2=0x7f3e4c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.644+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 0x7f3e4c0a3ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3e68108630 tx=0x7f3e58005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:49.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.644+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 msgr2=0x7f3e681a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.644+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 0x7f3e681a4c00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d680 tx=0x7f3e6000db40 comp rx=0 tx=0).stop 2024-03-11T18:30:49.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 shutdown_connections 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3e4c0a1a30 0x7f3e4c0a3ef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e68109190 0x7f3e681a4c00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e681087d0 0x7f3e6819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 --2- 172.21.15.2:0/2493273705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e681075e0 0x7f3e6819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 >> 172.21.15.2:0/2493273705 conn(0x7f3e68076570 msgr2=0x7f3e6810bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 shutdown_connections 2024-03-11T18:30:49.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.645+0000 7f3e6e200640 1 -- 172.21.15.2:0/2493273705 wait complete. 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:49 smithi023 ceph-mon[36406]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:49 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2355715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:49 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1439101417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:49 smithi165 ceph-mon[36634]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:49 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2355715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:49 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1439101417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:30:49.666 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 -- 172.21.15.2:0/840081619 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c104d90 msgr2=0x7f863c10b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 --2- 172.21.15.2:0/840081619 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c104d90 0x7f863c10b620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8634009ca0 tx=0x7f863402c0f0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 -- 172.21.15.2:0/840081619 shutdown_connections 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 --2- 172.21.15.2:0/840081619 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c104d90 0x7f863c10b620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 --2- 172.21.15.2:0/840081619 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1043d0 0x7f863c104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 --2- 172.21.15.2:0/840081619 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c1031e0 0x7f863c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 -- 172.21.15.2:0/840081619 >> 172.21.15.2:0/840081619 conn(0x7f863c0fe750 msgr2=0x7f863c100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:49.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 -- 172.21.15.2:0/840081619 shutdown_connections 2024-03-11T18:30:49.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.901+0000 7f8642200640 1 -- 172.21.15.2:0/840081619 wait complete. 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 Processor -- start 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 -- start start 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c1043d0 0x7f863c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 0x7f863c1a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c1a5150 con 0x7f863c1031e0 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c1a52c0 con 0x7f863c104d90 2024-03-11T18:30:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.903+0000 7f8642200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c1a5480 con 0x7f863c1043d0 2024-03-11T18:30:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48202/0 (socket says 172.21.15.2:48202) 2024-03-11T18:30:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 -- 172.21.15.2:0/3900143065 learned_addr learned my addr 172.21.15.2:0/3900143065 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8641800640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 0x7f863c1a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f863be00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c1043d0 0x7f863c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 -- 172.21.15.2:0/3900143065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c1043d0 msgr2=0x7f863c19dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c1043d0 0x7f863c19dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 -- 172.21.15.2:0/3900143065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 msgr2=0x7f863c1a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 0x7f863c1a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8640e00640 1 -- 172.21.15.2:0/3900143065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634009900 con 0x7f863c1031e0 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.904+0000 7f8641800640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 0x7f863c1a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8640e00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f863000e9c0 tx=0x7f863000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630009700 con 0x7f863c1031e0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f863000cd80 con 0x7f863c1031e0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c1a5760 con 0x7f863c1031e0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630010430 con 0x7f863c1031e0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.905+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863c1a5c70 con 0x7f863c1031e0 2024-03-11T18:30:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.906+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863c10ebb0 con 0x7f863c1031e0 2024-03-11T18:30:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.909+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f8630003680 con 0x7f863c1031e0 2024-03-11T18:30:49.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.910+0000 7f8639600640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 0x7f86240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:49.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.910+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8630014070 con 0x7f863c1031e0 2024-03-11T18:30:49.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.911+0000 7f863be00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 0x7f86240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:49.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.911+0000 7f863be00640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 0x7f86240a3ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f863c19eb70 tx=0x7f8628005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:49.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:49.914+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86300969c0 con 0x7f863c1031e0 2024-03-11T18:30:50.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.115+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f863c1a5ef0 con 0x7f863c1031e0 2024-03-11T18:30:50.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.115+0000 7f8639600640 1 -- 172.21.15.2:0/3900143065 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f863009a080 con 0x7f863c1031e0 2024-03-11T18:30:50.117 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:30:50.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 msgr2=0x7f86240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 0x7f86240a3ef0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f863c19eb70 tx=0x7f8628005f90 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 msgr2=0x7f863c19d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f863000e9c0 tx=0x7f863000ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 shutdown_connections 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f86240a1a30 0x7f86240a3ef0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f863c104d90 0x7f863c1a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f863c1043d0 0x7f863c19dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 --2- 172.21.15.2:0/3900143065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f863c1031e0 0x7f863c19d660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 >> 172.21.15.2:0/3900143065 conn(0x7f863c0fe750 msgr2=0x7f863c0775d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 shutdown_connections 2024-03-11T18:30:50.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.118+0000 7f8642200640 1 -- 172.21.15.2:0/3900143065 wait complete. 2024-03-11T18:30:50.141 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T18:30:50.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.360+0000 7f3233e00640 1 -- 172.21.15.2:0/1388556277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234104380 msgr2=0x7f3234106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.360+0000 7f3233e00640 1 --2- 172.21.15.2:0/1388556277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234104380 0x7f3234106770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3220009900 tx=0x7f322002bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 -- 172.21.15.2:0/1388556277 shutdown_connections 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 --2- 172.21.15.2:0/1388556277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234106cb0 0x7f3234109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 --2- 172.21.15.2:0/1388556277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234104380 0x7f3234106770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 --2- 172.21.15.2:0/1388556277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234101a50 0x7f3234103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 -- 172.21.15.2:0/1388556277 >> 172.21.15.2:0/1388556277 conn(0x7f32340fb460 msgr2=0x7f32340fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 -- 172.21.15.2:0/1388556277 shutdown_connections 2024-03-11T18:30:50.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.361+0000 7f3233e00640 1 -- 172.21.15.2:0/1388556277 wait complete. 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.362+0000 7f3233e00640 1 Processor -- start 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.362+0000 7f3233e00640 1 -- start start 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.362+0000 7f3233e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 0x7f3234197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234106cb0 0x7f323419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323419eb20 con 0x7f3234104380 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323419ec90 con 0x7f3234106cb0 2024-03-11T18:30:50.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323419ee50 con 0x7f3234101a50 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 0x7f3234197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41792/0 (socket says 172.21.15.2:41792) 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232a00640 1 -- 172.21.15.2:0/3296148757 learned_addr learned my addr 172.21.15.2:0/3296148757 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3233400640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234106cb0 0x7f323419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232a00640 1 -- 172.21.15.2:0/3296148757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234106cb0 msgr2=0x7f323419e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.363+0000 7f3232a00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234106cb0 0x7f323419e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3232a00640 1 -- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 msgr2=0x7f3234197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3232a00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 0x7f3234197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3232a00640 1 -- 172.21.15.2:0/3296148757 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220009560 con 0x7f3234101a50 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3232000640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 0x7f3234197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:50.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3232a00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f322400d950 tx=0x7f322400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:50.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224009a30 con 0x7f3234101a50 2024-03-11T18:30:50.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3224009b90 con 0x7f3234101a50 2024-03-11T18:30:50.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323419f130 con 0x7f3234101a50 2024-03-11T18:30:50.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224004a00 con 0x7f3234101a50 2024-03-11T18:30:50.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.364+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323419f610 con 0x7f3234101a50 2024-03-11T18:30:50.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.366+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32340fcf50 con 0x7f3234101a50 2024-03-11T18:30:50.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.367+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3224009d00 con 0x7f3234101a50 2024-03-11T18:30:50.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.369+0000 7f322b400640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 0x7f32180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:50.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.369+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f32240cd020 con 0x7f3234101a50 2024-03-11T18:30:50.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.369+0000 7f3232000640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 0x7f32180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:50.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.370+0000 7f3232000640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 0x7f32180a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3220005ab0 tx=0x7f3220005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:50.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.373+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3224095ff0 con 0x7f3234101a50 2024-03-11T18:30:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:50 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2493273705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:50 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3900143065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.589+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f323410c9c0 con 0x7f3234101a50 2024-03-11T18:30:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.590+0000 7f322b400640 1 -- 172.21.15.2:0/3296148757 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) v1 ==== 81+0+581 (secure 0 0 0) 0x7f3224017260 con 0x7f3234101a50 2024-03-11T18:30:50.591 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:50.591 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:50.591 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1969024355 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:50.591 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3822870648 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/247107605 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3920276264 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3359009707 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3303291273 2024-03-12T18:22:28.050188+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:30:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:listed 10 entries 2024-03-11T18:30:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 msgr2=0x7f32180a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 0x7f32180a3ef0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3220005ab0 tx=0x7f3220005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 msgr2=0x7f3234197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f322400d950 tx=0x7f322400de10 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 shutdown_connections 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f32180a1a30 0x7f32180a3ef0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3234106cb0 0x7f323419e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3234104380 0x7f3234197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 --2- 172.21.15.2:0/3296148757 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3234101a50 0x7f3234197030 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 >> 172.21.15.2:0/3296148757 conn(0x7f32340fb460 msgr2=0x7f3234102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 shutdown_connections 2024-03-11T18:30:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:50.593+0000 7f3233e00640 1 -- 172.21.15.2:0/3296148757 wait complete. 2024-03-11T18:30:50.614 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-03-11T18:30:50.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:50.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-11T18:30:50.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:50 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2493273705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:50 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3900143065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:50 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2493273705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:50 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3900143065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:30:50.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi002.stderr:ceph-fuse[76058]: fuse finished with error 0 and tester_r 0 2024-03-11T18:30:50.715 INFO:teuthology.orchestra.run:waiting for 300 2024-03-11T18:30:51.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:51 smithi165 ceph-mon[36634]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:51.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:51 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3296148757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:51.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:51 smithi023 ceph-mon[36406]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:51.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:51 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3296148757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:51.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:51 smithi002 ceph-mon[31328]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:51.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:51 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3296148757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:30:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:53 smithi165 ceph-mon[36634]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:53.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:53 smithi023 ceph-mon[36406]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:53.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:53 smithi002 ceph-mon[31328]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:55.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:55 smithi165 ceph-mon[36634]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:55.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:55 smithi023 ceph-mon[36406]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:55.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:55 smithi002 ceph-mon[31328]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:56.817 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:30:56.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:30:56.817 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-11T18:30:56.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 -- 172.21.15.2:0/3691804429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244101210 msgr2=0x7f1244101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 --2- 172.21.15.2:0/3691804429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244101210 0x7f1244101630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f123c009860 tx=0x7f123c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 -- 172.21.15.2:0/3691804429 shutdown_connections 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 --2- 172.21.15.2:0/3691804429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 0x7f1244109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 --2- 172.21.15.2:0/3691804429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244102400 0x7f1244102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 --2- 172.21.15.2:0/3691804429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244101210 0x7f1244101630 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.114+0000 7f124c000640 1 -- 172.21.15.2:0/3691804429 >> 172.21.15.2:0/3691804429 conn(0x7f12440fc7a0 msgr2=0x7f12440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:57.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.115+0000 7f124c000640 1 -- 172.21.15.2:0/3691804429 shutdown_connections 2024-03-11T18:30:57.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.115+0000 7f124c000640 1 -- 172.21.15.2:0/3691804429 wait complete. 2024-03-11T18:30:57.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.116+0000 7f124c000640 1 Processor -- start 2024-03-11T18:30:57.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.116+0000 7f124c000640 1 -- start start 2024-03-11T18:30:57.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.116+0000 7f124c000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244101210 0x7f1244199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.116+0000 7f124c000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.116+0000 7f124c000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 0x7f12441a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f124c000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441a0f10 con 0x7f1244101210 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f124c000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441a1080 con 0x7f1244102400 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f124c000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12441a1240 con 0x7f1244102dc0 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1249600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244101210 0x7f1244199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47274/0 (socket says 172.21.15.2:47274) 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 -- 172.21.15.2:0/3841375168 learned_addr learned my addr 172.21.15.2:0/3841375168 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:57.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f124a000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 0x7f12441a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 -- 172.21.15.2:0/3841375168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 msgr2=0x7f12441a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 0x7f12441a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 -- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244101210 msgr2=0x7f1244199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244101210 0x7f1244199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.117+0000 7f1248c00640 1 -- 172.21.15.2:0/3841375168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1230009900 con 0x7f1244102400 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f1248c00640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f123000c420 tx=0x7f1230009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230003970 con 0x7f1244102400 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1230004020 con 0x7f1244102400 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c009490 con 0x7f1244102400 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123000ba10 con 0x7f1244102400 2024-03-11T18:30:57.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.118+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12441a18f0 con 0x7f1244102400 2024-03-11T18:30:57.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.119+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124410cbb0 con 0x7f1244102400 2024-03-11T18:30:57.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.122+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f123000a040 con 0x7f1244102400 2024-03-11T18:30:57.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.123+0000 7f1242000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 0x7f12280a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.124+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f12300cd880 con 0x7f1244102400 2024-03-11T18:30:57.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.124+0000 7f1249600640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 0x7f12280a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.124+0000 7f1249600640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 0x7f12280a3f70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f123c009830 tx=0x7f123c034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:57.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.128+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12300967f0 con 0x7f1244102400 2024-03-11T18:30:57.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.333+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124410cd40 con 0x7f1244102400 2024-03-11T18:30:57.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.335+0000 7f1242000640 1 -- 172.21.15.2:0/3841375168 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f1230004190 con 0x7f1244102400 2024-03-11T18:30:57.337 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:57.338 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 msgr2=0x7f12280a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 0x7f12280a3f70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f123c009830 tx=0x7f123c034c00 comp rx=0 tx=0).stop 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 msgr2=0x7f1244199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f123000c420 tx=0x7f1230009f90 comp rx=0 tx=0).stop 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 shutdown_connections 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f12280a1ab0 0x7f12280a3f70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1244102dc0 0x7f12441a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1244102400 0x7f1244199960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 --2- 172.21.15.2:0/3841375168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1244101210 0x7f1244199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.338+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 >> 172.21.15.2:0/3841375168 conn(0x7f12440fc7a0 msgr2=0x7f12440733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.339+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 shutdown_connections 2024-03-11T18:30:57.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.339+0000 7f124c000640 1 -- 172.21.15.2:0/3841375168 wait complete. 2024-03-11T18:30:57.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:30:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 -- 172.21.15.2:0/191767523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104490 msgr2=0x7f3c04104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 --2- 172.21.15.2:0/191767523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104490 0x7f3c04104910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009900 tx=0x7f3bec02bc50 comp rx=0 tx=0).stop 2024-03-11T18:30:57.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 -- 172.21.15.2:0/191767523 shutdown_connections 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 --2- 172.21.15.2:0/191767523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c04104e50 0x7f3c0410b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 --2- 172.21.15.2:0/191767523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104490 0x7f3c04104910 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 --2- 172.21.15.2:0/191767523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c041032a0 0x7f3c041036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 -- 172.21.15.2:0/191767523 >> 172.21.15.2:0/191767523 conn(0x7f3c040fe830 msgr2=0x7f3c04100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 -- 172.21.15.2:0/191767523 shutdown_connections 2024-03-11T18:30:57.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.586+0000 7f3c03e00640 1 -- 172.21.15.2:0/191767523 wait complete. 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.587+0000 7f3c03e00640 1 Processor -- start 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 -- start start 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04104490 0x7f3c041997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104e50 0x7f3c041a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a0da0 con 0x7f3c04104e50 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a0f10 con 0x7f3c04104490 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c041a10d0 con 0x7f3c041032a0 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c02a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c02a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48070/0 (socket says 172.21.15.2:48070) 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c02a00640 1 -- 172.21.15.2:0/959721671 learned_addr learned my addr 172.21.15.2:0/959721671 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3c03400640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104e50 0x7f3c041a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.588+0000 7f3bfa000640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04104490 0x7f3c041997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 -- 172.21.15.2:0/959721671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04104490 msgr2=0x7f3c041997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04104490 0x7f3c041997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 -- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104e50 msgr2=0x7f3c041a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104e50 0x7f3c041a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 -- 172.21.15.2:0/959721671 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4009490 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c02a00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400e9c0 tx=0x7f3bf400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400cde0 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bf4010910 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.589+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec009560 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.590+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4018a10 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.590+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c041a1700 con 0x7f3c041032a0 2024-03-11T18:30:57.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.591+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0410a560 con 0x7f3c041032a0 2024-03-11T18:30:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.592+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3bf4010a80 con 0x7f3c041032a0 2024-03-11T18:30:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.594+0000 7f3bfbe00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 0x7f3be80a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.594+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3bf4014070 con 0x7f3c041032a0 2024-03-11T18:30:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.594+0000 7f3bfa000640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 0x7f3be80a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.595+0000 7f3bfa000640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 0x7f3be80a3ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009230 tx=0x7f3bec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:57.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.598+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3bf4096ce0 con 0x7f3c041032a0 2024-03-11T18:30:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:57 smithi165 ceph-mon[36634]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T18:30:57.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:57 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3841375168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:57 smithi023 ceph-mon[36406]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T18:30:57.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:57 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3841375168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:57 smithi002 ceph-mon[31328]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-11T18:30:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:57 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3841375168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:57.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.799+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c0402cdd0 con 0x7f3c041032a0 2024-03-11T18:30:57.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.802+0000 7f3bfbe00640 1 -- 172.21.15.2:0/959721671 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f3bf409a3a0 con 0x7f3c041032a0 2024-03-11T18:30:57.804 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:30:57.804 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:30:57.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.804+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 msgr2=0x7f3be80a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.804+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 0x7f3be80a3ee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009230 tx=0x7f3bec005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.804+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 msgr2=0x7f3c041992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.804+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400e9c0 tx=0x7f3bf400ee80 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 shutdown_connections 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3be80a1a20 0x7f3be80a3ee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c04104e50 0x7f3c041a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c04104490 0x7f3c041997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 --2- 172.21.15.2:0/959721671 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c041032a0 0x7f3c041992b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 >> 172.21.15.2:0/959721671 conn(0x7f3c040fe830 msgr2=0x7f3c04107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:57.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 shutdown_connections 2024-03-11T18:30:57.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:57.805+0000 7f3c03e00640 1 -- 172.21.15.2:0/959721671 wait complete. 2024-03-11T18:30:57.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-11T18:30:58.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 -- 172.21.15.2:0/1186506841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41089b0 msgr2=0x7fc4c4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 --2- 172.21.15.2:0/1186506841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41089b0 0x7fc4c4108e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009900 tx=0x7fc4b002bc40 comp rx=0 tx=0).stop 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 -- 172.21.15.2:0/1186506841 shutdown_connections 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 --2- 172.21.15.2:0/1186506841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c4109370 0x7fc4c410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 --2- 172.21.15.2:0/1186506841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41089b0 0x7fc4c4108e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.049+0000 7fc4caa00640 1 --2- 172.21.15.2:0/1186506841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c41077c0 0x7fc4c4107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.050+0000 7fc4caa00640 1 -- 172.21.15.2:0/1186506841 >> 172.21.15.2:0/1186506841 conn(0x7fc4c4076770 msgr2=0x7fc4c4078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.050+0000 7fc4caa00640 1 -- 172.21.15.2:0/1186506841 shutdown_connections 2024-03-11T18:30:58.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.050+0000 7fc4caa00640 1 -- 172.21.15.2:0/1186506841 wait complete. 2024-03-11T18:30:58.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 Processor -- start 2024-03-11T18:30:58.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 -- start start 2024-03-11T18:30:58.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 0x7fc4c419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c4109370 0x7fc4c41a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41a52d0 con 0x7fc4c4109370 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41a5440 con 0x7fc4c41077c0 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.051+0000 7fc4caa00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41a5600 con 0x7fc4c41089b0 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c4109370 0x7fc4c41a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47308/0 (socket says 172.21.15.2:47308) 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3e00640 1 -- 172.21.15.2:0/3320731764 learned_addr learned my addr 172.21.15.2:0/3320731764 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:58.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 0x7fc4c419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 -- 172.21.15.2:0/3320731764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 msgr2=0x7fc4c419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 -- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c4109370 msgr2=0x7fc4c41a4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c4109370 0x7fc4c41a4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.052+0000 7fc4c3400640 1 -- 172.21.15.2:0/3320731764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b0009560 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4c3e00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4c3400640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 0x7fc4c419dd20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0000c00 tx=0x7fc4b0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0004df0 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4b0005430 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c41a5880 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.053+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b003e450 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.054+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c41a5d90 con 0x7fc4c41089b0 2024-03-11T18:30:58.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.055+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c410eae0 con 0x7fc4c41089b0 2024-03-11T18:30:58.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.057+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc4b0003680 con 0x7fc4c41089b0 2024-03-11T18:30:58.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.059+0000 7fc4c0c00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 0x7fc4a80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.059+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc4b00efa80 con 0x7fc4c41089b0 2024-03-11T18:30:58.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.059+0000 7fc4c3e00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 0x7fc4a80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.060+0000 7fc4c3e00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 0x7fc4a80a3ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:58.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.062+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc4b00b89f0 con 0x7fc4c41089b0 2024-03-11T18:30:58.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.306+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc4c402cdd0 con 0x7fc4c41089b0 2024-03-11T18:30:58.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.403+0000 7fc4c0c00640 1 -- 172.21.15.2:0/3320731764 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc4b00bc0b0 con 0x7fc4c41089b0 2024-03-11T18:30:58.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 msgr2=0x7fc4a80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 0x7fc4a80a3ef0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005fd0 tx=0x7fc4b400b4a0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 msgr2=0x7fc4c419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 0x7fc4c419dd20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0000c00 tx=0x7fc4b0004270 comp rx=0 tx=0).stop 2024-03-11T18:30:58.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 shutdown_connections 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4a80a1a30 0x7fc4a80a3ef0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4c4109370 0x7fc4c41a4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4c41089b0 0x7fc4c419dd20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 --2- 172.21.15.2:0/3320731764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4c41077c0 0x7fc4c419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.406+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 >> 172.21.15.2:0/3320731764 conn(0x7fc4c4076770 msgr2=0x7fc4c410bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.407+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 shutdown_connections 2024-03-11T18:30:58.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.407+0000 7fc4caa00640 1 -- 172.21.15.2:0/3320731764 wait complete. 2024-03-11T18:30:58.431 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-11T18:30:58.431 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-11T18:30:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 -- 172.21.15.2:0/4125988092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1031c0 msgr2=0x7efe5c1035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 --2- 172.21.15.2:0/4125988092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1031c0 0x7efe5c1035e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efe4c009900 tx=0x7efe4c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 -- 172.21.15.2:0/4125988092 shutdown_connections 2024-03-11T18:30:58.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 --2- 172.21.15.2:0/4125988092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c104d70 0x7efe5c10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 --2- 172.21.15.2:0/4125988092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c1043b0 0x7efe5c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 --2- 172.21.15.2:0/4125988092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1031c0 0x7efe5c1035e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 -- 172.21.15.2:0/4125988092 >> 172.21.15.2:0/4125988092 conn(0x7efe5c0fe750 msgr2=0x7efe5c100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 -- 172.21.15.2:0/4125988092 shutdown_connections 2024-03-11T18:30:58.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.657+0000 7efe61c00640 1 -- 172.21.15.2:0/4125988092 wait complete. 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 Processor -- start 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 -- start start 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c1031c0 0x7efe5c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 0x7efe5c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c1a0d80 con 0x7efe5c1031c0 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c1a0ef0 con 0x7efe5c1043b0 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.658+0000 7efe61c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c1a10b0 con 0x7efe5c104d70 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c1031c0 0x7efe5c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47330/0 (socket says 172.21.15.2:47330) 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 -- 172.21.15.2:0/2379861143 learned_addr learned my addr 172.21.15.2:0/2379861143 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe61200640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 0x7efe5c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 -- 172.21.15.2:0/2379861143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 msgr2=0x7efe5c1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 0x7efe5c1a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 -- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c1031c0 msgr2=0x7efe5c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c1031c0 0x7efe5c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 -- 172.21.15.2:0/2379861143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c009560 con 0x7efe5c1043b0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe61200640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 0x7efe5c1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.659+0000 7efe5b400640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe4800e9f0 tx=0x7efe4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.660+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48009700 con 0x7efe5c1043b0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.660+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe4800cd80 con 0x7efe5c1043b0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.660+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c1a1390 con 0x7efe5c1043b0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.660+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48018920 con 0x7efe5c1043b0 2024-03-11T18:30:58.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.660+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c1a18a0 con 0x7efe5c1043b0 2024-03-11T18:30:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.661+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c10a5b0 con 0x7efe5c1043b0 2024-03-11T18:30:58.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.664+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7efe48018a80 con 0x7efe5c1043b0 2024-03-11T18:30:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.665+0000 7efe58c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 0x7efe400a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:58.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.665+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7efe48014070 con 0x7efe5c1043b0 2024-03-11T18:30:58.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.665+0000 7efe5be00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 0x7efe400a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:58.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.666+0000 7efe5be00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 0x7efe400a3ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe4c005ab0 tx=0x7efe4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:58.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.666+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efe48096a00 con 0x7efe5c1043b0 2024-03-11T18:30:58.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:58 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/959721671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:58.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:58 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3320731764' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:58 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:58 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/959721671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:58 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3320731764' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:58 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:58 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/959721671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:30:58.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:58 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3320731764' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:58 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-11T18:30:58.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:58.908+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7efe5c10ede0 con 0x7efe5c1043b0 2024-03-11T18:30:59.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.414+0000 7efe58c00640 1 -- 172.21.15.2:0/2379861143 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7efe4809a0c0 con 0x7efe5c1043b0 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 msgr2=0x7efe400a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 0x7efe400a3ef0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efe4c005ab0 tx=0x7efe4c005fd0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 msgr2=0x7efe5c1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe4800e9f0 tx=0x7efe4800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 shutdown_connections 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe400a1a30 0x7efe400a3ef0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe5c104d70 0x7efe5c1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5c1043b0 0x7efe5c1997d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 --2- 172.21.15.2:0/2379861143 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe5c1031c0 0x7efe5c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.418+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 >> 172.21.15.2:0/2379861143 conn(0x7efe5c0fe750 msgr2=0x7efe5c107780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:59.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.419+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 shutdown_connections 2024-03-11T18:30:59.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.419+0000 7efe61c00640 1 -- 172.21.15.2:0/2379861143 wait complete. 2024-03-11T18:30:59.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-11T18:30:59.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 -- 172.21.15.2:0/16721359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1045a0 msgr2=0x7f632c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:59.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 --2- 172.21.15.2:0/16721359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1045a0 0x7f632c104a20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6314009900 tx=0x7f631402bc30 comp rx=0 tx=0).stop 2024-03-11T18:30:59.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 -- 172.21.15.2:0/16721359 shutdown_connections 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 --2- 172.21.15.2:0/16721359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 0x7f632c10b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 --2- 172.21.15.2:0/16721359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1045a0 0x7f632c104a20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 --2- 172.21.15.2:0/16721359 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1033b0 0x7f632c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.656+0000 7f6332800640 1 -- 172.21.15.2:0/16721359 >> 172.21.15.2:0/16721359 conn(0x7f632c0fe940 msgr2=0x7f632c100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.657+0000 7f6332800640 1 -- 172.21.15.2:0/16721359 shutdown_connections 2024-03-11T18:30:59.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.657+0000 7f6332800640 1 -- 172.21.15.2:0/16721359 wait complete. 2024-03-11T18:30:59.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.657+0000 7f6332800640 1 Processor -- start 2024-03-11T18:30:59.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 -- start start 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1045a0 0x7f632c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 0x7f632c1a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1a0f20 con 0x7f632c104f60 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1a1090 con 0x7f632c1033b0 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f6332800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1a1250 con 0x7f632c1045a0 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f632b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f632be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 0x7f632c1a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f632b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47344/0 (socket says 172.21.15.2:47344) 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f632b400640 1 -- 172.21.15.2:0/593262349 learned_addr learned my addr 172.21.15.2:0/593262349 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:30:59.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.658+0000 7f632aa00640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1045a0 0x7f632c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:59.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 -- 172.21.15.2:0/593262349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1045a0 msgr2=0x7f632c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:59.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1045a0 0x7f632c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 -- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 msgr2=0x7f632c1a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:30:59.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 0x7f632c1a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:30:59.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 -- 172.21.15.2:0/593262349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314009560 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.659+0000 7f632b400640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f631c00eb70 tx=0x7f631c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.660+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c003b70 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.660+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f631c00ee70 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.660+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c0107f0 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.660+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c1a1530 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.660+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1a1a10 con 0x7f632c1033b0 2024-03-11T18:30:59.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.661+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c10a720 con 0x7f632c1033b0 2024-03-11T18:30:59.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.663+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f631c004000 con 0x7f632c1033b0 2024-03-11T18:30:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.664+0000 7f6323e00640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 0x7f63100a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:30:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.664+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f631c014070 con 0x7f632c1033b0 2024-03-11T18:30:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.665+0000 7f632aa00640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 0x7f63100a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:30:59.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.665+0000 7f632aa00640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 0x7f63100a3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6314009230 tx=0x7f6314005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:30:59.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.666+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f631c096c20 con 0x7f632c1033b0 2024-03-11T18:30:59.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:59 smithi002 ceph-mon[31328]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:59.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:59 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T18:30:59.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:59 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:59.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:59 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2379861143' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:30:59 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:59 smithi165 ceph-mon[36634]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:59 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T18:30:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:59 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:59 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2379861143' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:30:59 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:59 smithi023 ceph-mon[36406]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:59 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:59 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:59 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2379861143' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:30:59 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-11T18:30:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:30:59.909+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f632c10ef40 con 0x7f632c1033b0 2024-03-11T18:31:00.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.437+0000 7f6323e00640 1 -- 172.21.15.2:0/593262349 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f631c003e60 con 0x7f632c1033b0 2024-03-11T18:31:00.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 msgr2=0x7f63100a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:00.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 0x7f63100a3f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6314009230 tx=0x7f6314005fd0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 msgr2=0x7f632c199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:00.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f631c00eb70 tx=0x7f631c00c6a0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 shutdown_connections 2024-03-11T18:31:00.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f63100a1ab0 0x7f63100a3f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c104f60 0x7f632c1a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f632c1045a0 0x7f632c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 --2- 172.21.15.2:0/593262349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f632c1033b0 0x7f632c199430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.443+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 >> 172.21.15.2:0/593262349 conn(0x7f632c0fe940 msgr2=0x7f632c1078f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:00.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.444+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 shutdown_connections 2024-03-11T18:31:00.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.444+0000 7f6332800640 1 -- 172.21.15.2:0/593262349 wait complete. 2024-03-11T18:31:00.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-11T18:31:00.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.686+0000 7fcc16e00640 1 -- 172.21.15.2:0/2306548503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 msgr2=0x7fcc10107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:00.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.686+0000 7fcc16e00640 1 --2- 172.21.15.2:0/2306548503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc10107be0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009900 tx=0x7fcc0002bc30 comp rx=0 tx=0).stop 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 -- 172.21.15.2:0/2306548503 shutdown_connections 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 --2- 172.21.15.2:0/2306548503 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc10109370 0x7fcc1010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 --2- 172.21.15.2:0/2306548503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc101089b0 0x7fcc10108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 --2- 172.21.15.2:0/2306548503 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc10107be0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 -- 172.21.15.2:0/2306548503 >> 172.21.15.2:0/2306548503 conn(0x7fcc10076770 msgr2=0x7fcc10078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 -- 172.21.15.2:0/2306548503 shutdown_connections 2024-03-11T18:31:00.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.687+0000 7fcc16e00640 1 -- 172.21.15.2:0/2306548503 wait complete. 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 Processor -- start 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 -- start start 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc1019d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc101089b0 0x7fcc1019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:00.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc101a53b0 con 0x7fcc101077c0 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc101a5520 con 0x7fcc10109370 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc16e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc101a56e0 con 0x7fcc101089b0 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.688+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc1019d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47354/0 (socket says 172.21.15.2:47354) 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 -- 172.21.15.2:0/3175160465 learned_addr learned my addr 172.21.15.2:0/3175160465 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc0f400640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc101089b0 0x7fcc1019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 -- 172.21.15.2:0/3175160465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc101089b0 msgr2=0x7fcc1019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc101089b0 0x7fcc1019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 -- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 msgr2=0x7fcc1019d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc1019d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:00.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 -- 172.21.15.2:0/3175160465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc00009560 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc14e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800eb10 tx=0x7fcc0800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.689+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08004b20 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.690+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc08004c80 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.690+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08010650 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.690+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc101a59c0 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.690+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc101a5ed0 con 0x7fcc10109370 2024-03-11T18:31:00.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.691+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1010eb30 con 0x7fcc10109370 2024-03-11T18:31:00.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.693+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fcc08010810 con 0x7fcc10109370 2024-03-11T18:31:00.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.694+0000 7fcc0cc00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 0x7fcbf40a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:00.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.694+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcc08014070 con 0x7fcc10109370 2024-03-11T18:31:00.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.695+0000 7fcc0fe00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 0x7fcbf40a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.695+0000 7fcc0fe00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 0x7fcbf40a3f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcc0002c240 tx=0x7fcc0002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:00.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.696+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc08096cf0 con 0x7fcc10109370 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T18:31:00.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T18:31:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/593262349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:00 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/593262349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:00 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T18:31:00.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-11T18:31:00.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-11T18:31:00.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-11T18:31:00.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/593262349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.907 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:00 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T18:31:00.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:00.941+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fcc101131e0 con 0x7fcc10109370 2024-03-11T18:31:01.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.453+0000 7fcc0cc00640 1 -- 172.21.15.2:0/3175160465 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcc0809a3b0 con 0x7fcc10109370 2024-03-11T18:31:01.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.457+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 msgr2=0x7fcbf40a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 0x7fcbf40a3f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcc0002c240 tx=0x7fcc0002c7e0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 msgr2=0x7fcc101a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800eb10 tx=0x7fcc0800c2d0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 shutdown_connections 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcbf40a1ab0 0x7fcbf40a3f70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc10109370 0x7fcc101a4e70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc101089b0 0x7fcc1019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 --2- 172.21.15.2:0/3175160465 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc101077c0 0x7fcc1019d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 >> 172.21.15.2:0/3175160465 conn(0x7fcc10076770 msgr2=0x7fcc1010bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 shutdown_connections 2024-03-11T18:31:01.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.458+0000 7fcc16e00640 1 -- 172.21.15.2:0/3175160465 wait complete. 2024-03-11T18:31:01.480 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-11T18:31:01.480 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 -- 172.21.15.2:0/3798749812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 msgr2=0x7f29cc102630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3798749812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc102630 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009860 tx=0x7f29b402bb70 comp rx=0 tx=0).stop 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 -- 172.21.15.2:0/3798749812 shutdown_connections 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3798749812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc102b70 0x7f29cc109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3798749812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc102630 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3798749812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc100fc0 0x7f29cc1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 -- 172.21.15.2:0/3798749812 >> 172.21.15.2:0/3798749812 conn(0x7f29cc0fc530 msgr2=0x7f29cc0fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 -- 172.21.15.2:0/3798749812 shutdown_connections 2024-03-11T18:31:01.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.710+0000 7f29cbe00640 1 -- 172.21.15.2:0/3798749812 wait complete. 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.711+0000 7f29cbe00640 1 Processor -- start 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.711+0000 7f29cbe00640 1 -- start start 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc102b70 0x7f29cc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19eb30 con 0x7f29cc102b70 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19eca0 con 0x7f29cc100fc0 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cbe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc19ee60 con 0x7f29cc1021b0 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:01.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47368/0 (socket says 172.21.15.2:47368) 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29cb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc102b70 0x7f29cc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 -- 172.21.15.2:0/3462088550 learned_addr learned my addr 172.21.15.2:0/3462088550 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29c2000640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 -- 172.21.15.2:0/3462088550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 msgr2=0x7f29cc197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 -- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc102b70 msgr2=0x7f29cc19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.712+0000 7f29caa00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc102b70 0x7f29cc19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29caa00640 1 -- 172.21.15.2:0/3462088550 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4009490 con 0x7f29cc100fc0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29c2000640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29caa00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00ebc0 tx=0x7f29bc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc003970 con 0x7f29cc100fc0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29bc004030 con 0x7f29cc100fc0 2024-03-11T18:31:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc019720 con 0x7f29cc100fc0 2024-03-11T18:31:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc19f140 con 0x7f29cc100fc0 2024-03-11T18:31:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.713+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc19f5d0 con 0x7f29cc100fc0 2024-03-11T18:31:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.714+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc108360 con 0x7f29cc100fc0 2024-03-11T18:31:01.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.717+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f29bc0044f0 con 0x7f29cc100fc0 2024-03-11T18:31:01.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.718+0000 7f29c3e00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 0x7f29b00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:01.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.718+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f29bc014070 con 0x7f29cc100fc0 2024-03-11T18:31:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.718+0000 7f29c2000640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 0x7f29b00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:01.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.719+0000 7f29c2000640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 0x7f29b00a3f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009830 tx=0x7f29b4034be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:01.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.721+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f29bc096e90 con 0x7f29cc100fc0 2024-03-11T18:31:01.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-11T18:31:01.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:active 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: Cluster is now healthy 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3175160465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-11T18:31:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T18:31:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:active 2024-03-11T18:31:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T18:31:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: Cluster is now healthy 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3175160465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:01 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-11T18:31:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] up:active 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: Cluster is now healthy 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3175160465' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:01 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-11T18:31:01.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.951+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f29cc02cf20 con 0x7f29b00a1ab0 2024-03-11T18:31:01.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.993+0000 7f29c3e00640 1 -- 172.21.15.2:0/3462088550 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f29cc02cf20 con 0x7f29b00a1ab0 2024-03-11T18:31:01.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 msgr2=0x7f29b00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 0x7f29b00a3f70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009830 tx=0x7f29b4034be0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 msgr2=0x7f29cc197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00ebc0 tx=0x7f29bc00c2d0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 shutdown_connections 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f29b00a1ab0 0x7f29b00a3f70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29cc102b70 0x7f29cc19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f29cc1021b0 0x7f29cc197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 --2- 172.21.15.2:0/3462088550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29cc100fc0 0x7f29cc197040 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 >> 172.21.15.2:0/3462088550 conn(0x7f29cc0fc530 msgr2=0x7f29cc105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 shutdown_connections 2024-03-11T18:31:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:01.996+0000 7f29cbe00640 1 -- 172.21.15.2:0/3462088550 wait complete. 2024-03-11T18:31:02.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 -- 172.21.15.2:0/2615072370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174073b60 msgr2=0x7f6174073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 --2- 172.21.15.2:0/2615072370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174073b60 0x7f6174073fc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6160009900 tx=0x7f616002bcc0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 -- 172.21.15.2:0/2615072370 shutdown_connections 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 --2- 172.21.15.2:0/2615072370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174108e70 0x7f617410b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 --2- 172.21.15.2:0/2615072370 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174073b60 0x7f6174073fc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 --2- 172.21.15.2:0/2615072370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 -- 172.21.15.2:0/2615072370 >> 172.21.15.2:0/2615072370 conn(0x7f61740fc530 msgr2=0x7f61740fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:02.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 -- 172.21.15.2:0/2615072370 shutdown_connections 2024-03-11T18:31:02.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.214+0000 7f6178a00640 1 -- 172.21.15.2:0/2615072370 wait complete. 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.215+0000 7f6178a00640 1 Processor -- start 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.215+0000 7f6178a00640 1 -- start start 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174073b60 0x7f6174199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174108e70 0x7f61741a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741a0d10 con 0x7f6174075140 2024-03-11T18:31:02.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741a0e80 con 0x7f6174073b60 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6178a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741a1040 con 0x7f6174108e70 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39602/0 (socket says 172.21.15.2:39602) 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 -- 172.21.15.2:0/779749847 learned_addr learned my addr 172.21.15.2:0/779749847 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6172800640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174073b60 0x7f6174199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6173200640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174108e70 0x7f61741a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 -- 172.21.15.2:0/779749847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174108e70 msgr2=0x7f61741a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174108e70 0x7f61741a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 -- 172.21.15.2:0/779749847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174073b60 msgr2=0x7f6174199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174073b60 0x7f6174199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.216+0000 7f6171e00640 1 -- 172.21.15.2:0/779749847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6160009560 con 0x7f6174075140 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f6171e00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6160006010 tx=0x7f61600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:02.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61600044e0 con 0x7f6174075140 2024-03-11T18:31:02.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61600054e0 con 0x7f6174075140 2024-03-11T18:31:02.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616003e450 con 0x7f6174075140 2024-03-11T18:31:02.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61741a12c0 con 0x7f6174075140 2024-03-11T18:31:02.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.217+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741a1820 con 0x7f6174075140 2024-03-11T18:31:02.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.219+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6174107c30 con 0x7f6174075140 2024-03-11T18:31:02.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.221+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f61600035f0 con 0x7f6174075140 2024-03-11T18:31:02.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.223+0000 7f616b400640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 0x7f61580a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.223+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f61600f05f0 con 0x7f6174075140 2024-03-11T18:31:02.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.223+0000 7f6172800640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 0x7f61580a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.224+0000 7f6172800640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 0x7f61580a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f616400bbe0 tx=0x7f616400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:02.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.225+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f61600b9560 con 0x7f6174075140 2024-03-11T18:31:02.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:31:01.965+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:31:02.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:31:01.965+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:31:02.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:31:01.965+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:31:02.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:31:01.965+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:31:02.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:31:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:31:01.965+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:31:02.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.445+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f61740615c0 con 0x7f61580a1ab0 2024-03-11T18:31:02.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.489+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f61600b9930 con 0x7f6174075140 2024-03-11T18:31:02.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.491+0000 7f616b400640 1 -- 172.21.15.2:0/779749847 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f61740615c0 con 0x7f61580a1ab0 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 msgr2=0x7f61580a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 0x7f61580a3f70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f616400bbe0 tx=0x7f616400b390 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 msgr2=0x7f6174199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6160006010 tx=0x7f61600042f0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 shutdown_connections 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f61580a1ab0 0x7f61580a3f70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6174108e70 0x7f61741a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6174075140 0x7f6174199760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 --2- 172.21.15.2:0/779749847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6174073b60 0x7f6174199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 >> 172.21.15.2:0/779749847 conn(0x7f61740fc530 msgr2=0x7f6174104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:02.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 shutdown_connections 2024-03-11T18:31:02.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.494+0000 7f6178a00640 1 -- 172.21.15.2:0/779749847 wait complete. 2024-03-11T18:31:02.512 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 -- 172.21.15.2:0/4044412498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01031c0 msgr2=0x7f7ff01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/4044412498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01031c0 0x7f7ff01035e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0009900 tx=0x7f7fe002bc30 comp rx=0 tx=0).stop 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 -- 172.21.15.2:0/4044412498 shutdown_connections 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/4044412498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff0104d70 0x7f7ff010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/4044412498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff01043b0 0x7f7ff0104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/4044412498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01031c0 0x7f7ff01035e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 -- 172.21.15.2:0/4044412498 >> 172.21.15.2:0/4044412498 conn(0x7f7ff00fe750 msgr2=0x7f7ff0100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 -- 172.21.15.2:0/4044412498 shutdown_connections 2024-03-11T18:31:02.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.711+0000 7f7ff5a00640 1 -- 172.21.15.2:0/4044412498 wait complete. 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 Processor -- start 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 -- start start 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01043b0 0x7f7ff01997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff0104d70 0x7f7ff01a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff01a0da0 con 0x7f7ff01043b0 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff01a0f10 con 0x7f7ff01031c0 2024-03-11T18:31:02.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7ff5a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff01a10d0 con 0x7f7ff0104d70 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.712+0000 7f7fefe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fef400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01043b0 0x7f7ff01997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47410/0 (socket says 172.21.15.2:47410) 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 -- 172.21.15.2:0/2806394650 learned_addr learned my addr 172.21.15.2:0/2806394650 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7ff5000640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff0104d70 0x7f7ff01a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 -- 172.21.15.2:0/2806394650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff0104d70 msgr2=0x7f7ff01a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff0104d70 0x7f7ff01a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 -- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01043b0 msgr2=0x7f7ff01997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01043b0 0x7f7ff01997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 -- 172.21.15.2:0/2806394650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0009560 con 0x7f7ff01031c0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fefe00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0005ab0 tx=0x7f7fe0005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.713+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe00043d0 con 0x7f7ff01031c0 2024-03-11T18:31:02.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.714+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff01a1350 con 0x7f7ff01031c0 2024-03-11T18:31:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.714+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fe00052d0 con 0x7f7ff01031c0 2024-03-11T18:31:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.714+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0003d40 con 0x7f7ff01031c0 2024-03-11T18:31:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.714+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff01a1860 con 0x7f7ff01031c0 2024-03-11T18:31:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.715+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff010a580 con 0x7f7ff01031c0 2024-03-11T18:31:02.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.717+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f7fe0005440 con 0x7f7ff01031c0 2024-03-11T18:31:02.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.718+0000 7f7fecc00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 0x7f7fd40a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:02.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.718+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7fe00efa90 con 0x7f7ff01031c0 2024-03-11T18:31:02.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.718+0000 7f7fef400640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 0x7f7fd40a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:02.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.719+0000 7f7fef400640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 0x7f7fd40a3f80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ff019a7c0 tx=0x7f7fdc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:02.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.721+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7fe00b89b0 con 0x7f7ff01031c0 2024-03-11T18:31:02.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:02 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T18:31:02.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:02 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:active 2024-03-11T18:31:02.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:02 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-11T18:31:02.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:02 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:31:02.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:02 smithi002 ceph-mon[31328]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:02.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:02 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T18:31:02.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:02 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:active 2024-03-11T18:31:02.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:02 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-11T18:31:02.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:02 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:31:02.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:02 smithi165 ceph-mon[36634]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:02 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-11T18:31:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:02 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] up:active 2024-03-11T18:31:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:02 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-11T18:31:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:02 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:31:02.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:02 smithi023 ceph-mon[36406]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:02.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.953+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff002ce40 con 0x7f7fd40a1ac0 2024-03-11T18:31:02.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.961+0000 7f7fecc00640 1 -- 172.21.15.2:0/2806394650 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f7ff002ce40 con 0x7f7fd40a1ac0 2024-03-11T18:31:02.963 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/qa/sv_0/2fbf7596-03be-4369-8b74-8d0284ad5f65 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.963+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 msgr2=0x7f7fd40a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.963+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 0x7f7fd40a3f80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ff019a7c0 tx=0x7f7fdc005f90 comp rx=0 tx=0).stop 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.963+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 msgr2=0x7f7ff01992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.963+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0005ab0 tx=0x7f7fe0005e50 comp rx=0 tx=0).stop 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 shutdown_connections 2024-03-11T18:31:02.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7fd40a1ac0 0x7f7fd40a3f80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ff0104d70 0x7f7ff01a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ff01043b0 0x7f7ff01997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 --2- 172.21.15.2:0/2806394650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ff01031c0 0x7f7ff01992b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 >> 172.21.15.2:0/2806394650 conn(0x7f7ff00fe750 msgr2=0x7f7ff0107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 shutdown_connections 2024-03-11T18:31:02.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:02.964+0000 7f7ff5a00640 1 -- 172.21.15.2:0/2806394650 wait complete. 2024-03-11T18:31:02.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-11T18:31:03.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.202+0000 7feef4400640 1 -- 172.21.15.2:0/1749502948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec104f60 msgr2=0x7feeec10b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.202+0000 7feef4400640 1 --2- 172.21.15.2:0/1749502948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec104f60 0x7feeec10b7f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feee4009ca0 tx=0x7feee402c100 comp rx=0 tx=0).stop 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 -- 172.21.15.2:0/1749502948 shutdown_connections 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 --2- 172.21.15.2:0/1749502948 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec104f60 0x7feeec10b7f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 --2- 172.21.15.2:0/1749502948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec1045a0 0x7feeec104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 --2- 172.21.15.2:0/1749502948 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 0x7feeec1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 -- 172.21.15.2:0/1749502948 >> 172.21.15.2:0/1749502948 conn(0x7feeec0fe940 msgr2=0x7feeec100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 -- 172.21.15.2:0/1749502948 shutdown_connections 2024-03-11T18:31:03.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.203+0000 7feef4400640 1 -- 172.21.15.2:0/1749502948 wait complete. 2024-03-11T18:31:03.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 Processor -- start 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 -- start start 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 0x7feeec1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec1045a0 0x7feeec199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec1a0ec0 con 0x7feeec1045a0 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec1a1030 con 0x7feeec104f60 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.204+0000 7feef4400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec1a11f0 con 0x7feeec1033b0 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec1045a0 0x7feeec199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47414/0 (socket says 172.21.15.2:47414) 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 -- 172.21.15.2:0/545036771 learned_addr learned my addr 172.21.15.2:0/545036771 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef1a00640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 0x7feeec1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 -- 172.21.15.2:0/545036771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 msgr2=0x7feeec1993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 0x7feeec1993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 -- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec1045a0 msgr2=0x7feeec199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec1045a0 0x7feeec199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 -- 172.21.15.2:0/545036771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8009490 con 0x7feeec104f60 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.205+0000 7feef2400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feee402c640 tx=0x7feee4032fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.206+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4034640 con 0x7feeec104f60 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.206+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feee4003e50 con 0x7feeec104f60 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.206+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee4004840 con 0x7feeec104f60 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.206+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee4009900 con 0x7feeec104f60 2024-03-11T18:31:03.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.206+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec1a1740 con 0x7feeec104f60 2024-03-11T18:31:03.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.207+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec10a6d0 con 0x7feeec104f60 2024-03-11T18:31:03.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.209+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7feee400a040 con 0x7feeec104f60 2024-03-11T18:31:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.210+0000 7feeea000640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 0x7feed00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.210+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7feee40effe0 con 0x7feeec104f60 2024-03-11T18:31:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.210+0000 7feef1a00640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 0x7feed00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.211+0000 7feef1a00640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 0x7feed00a4000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feeec104400 tx=0x7feedc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:03.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.215+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feee40b8f40 con 0x7feeec104f60 2024-03-11T18:31:03.286 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: 2024-03-11T18:31:03.032+0000 7fe52f1b5700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-11T18:31:03.286 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: 2024-03-11T18:31:03.032+0000 7fe52f1b5700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-11T18:31:03.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.451+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7feeec02ceb0 con 0x7feed00a1b40 2024-03-11T18:31:03.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.479+0000 7feeea000640 1 -- 172.21.15.2:0/545036771 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feeec02ceb0 con 0x7feed00a1b40 2024-03-11T18:31:03.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.482+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 msgr2=0x7feed00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.482+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 0x7feed00a4000 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feeec104400 tx=0x7feedc005f90 comp rx=0 tx=0).stop 2024-03-11T18:31:03.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.482+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 msgr2=0x7feeec1a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.482+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7feee402c640 tx=0x7feee4032fc0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.482+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 shutdown_connections 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7feed00a1b40 0x7feed00a4000 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feeec104f60 0x7feeec1a0980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feeec1045a0 0x7feeec199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 --2- 172.21.15.2:0/545036771 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec1033b0 0x7feeec1993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 >> 172.21.15.2:0/545036771 conn(0x7feeec0fe940 msgr2=0x7feeec1078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 shutdown_connections 2024-03-11T18:31:03.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.483+0000 7feef4400640 1 -- 172.21.15.2:0/545036771 wait complete. 2024-03-11T18:31:03.505 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-11T18:31:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-mon[36406]: pgmap v338: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 682 B/s wr, 3 op/s 2024-03-11T18:31:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-mon[36406]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-mon[36406]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T18:31:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:03 smithi023 ceph-mon[36406]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 -- 172.21.15.2:0/2133680348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8109190 msgr2=0x7fc4d810fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 --2- 172.21.15.2:0/2133680348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8109190 0x7fc4d810fa20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009be0 tx=0x7fc4d002bef0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 -- 172.21.15.2:0/2133680348 shutdown_connections 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 --2- 172.21.15.2:0/2133680348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d8109190 0x7fc4d810fa20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 --2- 172.21.15.2:0/2133680348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d81087d0 0x7fc4d8108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 --2- 172.21.15.2:0/2133680348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 0x7fc4d8107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 -- 172.21.15.2:0/2133680348 >> 172.21.15.2:0/2133680348 conn(0x7fc4d8076570 msgr2=0x7fc4d80789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 -- 172.21.15.2:0/2133680348 shutdown_connections 2024-03-11T18:31:03.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.738+0000 7fc4de400640 1 -- 172.21.15.2:0/2133680348 wait complete. 2024-03-11T18:31:03.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 Processor -- start 2024-03-11T18:31:03.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 -- start start 2024-03-11T18:31:03.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 0x7fc4d819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d8109190 0x7fc4d81a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81a50e0 con 0x7fc4d8109190 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81a5250 con 0x7fc4d81087d0 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.739+0000 7fc4de400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81a5410 con 0x7fc4d81075e0 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47432/0 (socket says 172.21.15.2:47432) 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 -- 172.21.15.2:0/3173273855 learned_addr learned my addr 172.21.15.2:0/3173273855 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4dda00640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d8109190 0x7fc4d81a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4dd000640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 0x7fc4d819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 -- 172.21.15.2:0/3173273855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 msgr2=0x7fc4d819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 0x7fc4d819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 -- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d8109190 msgr2=0x7fc4d81a4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d8109190 0x7fc4d81a4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 -- 172.21.15.2:0/3173273855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0009840 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.740+0000 7fc4d7e00640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400e9c0 tx=0x7fc4c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.741+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c400ccc0 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.741+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c400ce20 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.741+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d81a56f0 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.741+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4010430 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.741+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d81a5c50 con 0x7fc4d81087d0 2024-03-11T18:31:03.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.742+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d810e830 con 0x7fc4d81087d0 2024-03-11T18:31:03.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.745+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fc4c4004f80 con 0x7fc4d81087d0 2024-03-11T18:31:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.746+0000 7fc4d5600640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 0x7fc4c00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.746+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc4c4014070 con 0x7fc4d81087d0 2024-03-11T18:31:03.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.746+0000 7fc4dd000640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 0x7fc4c00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:03.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.747+0000 7fc4dd000640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 0x7fc4c00a4000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8108630 tx=0x7fc4cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:03.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.750+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc4c4096890 con 0x7fc4d81087d0 2024-03-11T18:31:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:03 smithi002 ceph-mon[31328]: pgmap v338: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 682 B/s wr, 3 op/s 2024-03-11T18:31:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:03 smithi002 ceph-mon[31328]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:03 smithi002 ceph-mon[31328]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T18:31:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:03 smithi002 ceph-mon[31328]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:03 smithi165 ceph-mon[36634]: pgmap v338: 129 pgs: 129 active+clean; 583 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 682 B/s wr, 3 op/s 2024-03-11T18:31:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:03 smithi165 ceph-mon[36634]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:03 smithi165 ceph-mon[36634]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-11T18:31:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:03 smithi165 ceph-mon[36634]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:03.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.983+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4d802ceb0 con 0x7fc4c00a1b40 2024-03-11T18:31:03.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.992+0000 7fc4d5600640 1 -- 172.21.15.2:0/3173273855 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fc4d802ceb0 con 0x7fc4c00a1b40 2024-03-11T18:31:03.993 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/qa/sv_1/9e52e265-fb66-4d5f-95fb-56794f10b175 2024-03-11T18:31:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.994+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 msgr2=0x7fc4c00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.994+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 0x7fc4c00a4000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8108630 tx=0x7fc4cc005f90 comp rx=0 tx=0).stop 2024-03-11T18:31:03.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 msgr2=0x7fc4d819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400e9c0 tx=0x7fc4c400ee80 comp rx=0 tx=0).stop 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 shutdown_connections 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc4c00a1b40 0x7fc4c00a4000 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4d8109190 0x7fc4d81a4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4d81087d0 0x7fc4d819db30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 --2- 172.21.15.2:0/3173273855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc4d81075e0 0x7fc4d819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 >> 172.21.15.2:0/3173273855 conn(0x7fc4d8076570 msgr2=0x7fc4d810ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 shutdown_connections 2024-03-11T18:31:03.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:03.995+0000 7fc4de400640 1 -- 172.21.15.2:0/3173273855 wait complete. 2024-03-11T18:31:04.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:31:04.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.239+0000 7fc8c1e00640 1 -- 172.21.15.2:0/2752630398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1089b0 msgr2=0x7fc8bc108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.239+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/2752630398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1089b0 0x7fc8bc108e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009900 tx=0x7fc8a802bc30 comp rx=0 tx=0).stop 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 -- 172.21.15.2:0/2752630398 shutdown_connections 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/2752630398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc109370 0x7fc8bc10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/2752630398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1089b0 0x7fc8bc108e30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/2752630398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 -- 172.21.15.2:0/2752630398 >> 172.21.15.2:0/2752630398 conn(0x7fc8bc076780 msgr2=0x7fc8bc078ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 -- 172.21.15.2:0/2752630398 shutdown_connections 2024-03-11T18:31:04.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.240+0000 7fc8c1e00640 1 -- 172.21.15.2:0/2752630398 wait complete. 2024-03-11T18:31:04.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 Processor -- start 2024-03-11T18:31:04.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 -- start start 2024-03-11T18:31:04.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 0x7fc8bc19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1a3000 0x7fc8bc1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1a5930 con 0x7fc8bc1077c0 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1a5aa0 con 0x7fc8bc1a3000 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.241+0000 7fc8c1e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1a5c60 con 0x7fc8bc109370 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39684/0 (socket says 172.21.15.2:39684) 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 -- 172.21.15.2:0/1069052964 learned_addr learned my addr 172.21.15.2:0/1069052964 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8bb400640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1a3000 0x7fc8bc1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8ba000640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 0x7fc8bc19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 -- 172.21.15.2:0/1069052964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 msgr2=0x7fc8bc19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 0x7fc8bc19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 -- 172.21.15.2:0/1069052964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1a3000 msgr2=0x7fc8bc1a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1a3000 0x7fc8bc1a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 -- 172.21.15.2:0/1069052964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8009560 con 0x7fc8bc1077c0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8ba000640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 0x7fc8bc19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.242+0000 7fc8baa00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00e9c0 tx=0x7fc8ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.243+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0096e0 con 0x7fc8bc1077c0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.243+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8ac00cd80 con 0x7fc8bc1077c0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.243+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc1a5f40 con 0x7fc8bc1077c0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.243+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac010430 con 0x7fc8bc1077c0 2024-03-11T18:31:04.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.243+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc1a6360 con 0x7fc8bc1077c0 2024-03-11T18:31:04.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.244+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc10ec00 con 0x7fc8bc1077c0 2024-03-11T18:31:04.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.246+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fc8ac003680 con 0x7fc8bc1077c0 2024-03-11T18:31:04.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.247+0000 7fc8c0a00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 0x7fc8a00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.247+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc8ac014070 con 0x7fc8bc1077c0 2024-03-11T18:31:04.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.248+0000 7fc8ba000640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 0x7fc8a00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.248+0000 7fc8ba000640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 0x7fc8a00a4000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009230 tx=0x7fc8a8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:04.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.252+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8ac096a10 con 0x7fc8bc1077c0 2024-03-11T18:31:04.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.489+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8bc113470 con 0x7fc8bc1077c0 2024-03-11T18:31:04.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.492+0000 7fc8c0a00640 1 -- 172.21.15.2:0/1069052964 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7fc8ac09a0d0 con 0x7fc8bc1077c0 2024-03-11T18:31:04.494 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:04.494 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:31:04.494 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:31:04.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 msgr2=0x7fc8a00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 0x7fc8a00a4000 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009230 tx=0x7fc8a8005be0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 msgr2=0x7fc8bc19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00e9c0 tx=0x7fc8ac00ee80 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 shutdown_connections 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8a00a1b40 0x7fc8a00a4000 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8bc1a3000 0x7fc8bc1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8bc109370 0x7fc8bc19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 --2- 172.21.15.2:0/1069052964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8bc1077c0 0x7fc8bc19da60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 >> 172.21.15.2:0/1069052964 conn(0x7fc8bc076780 msgr2=0x7fc8bc10bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 shutdown_connections 2024-03-11T18:31:04.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.495+0000 7fc8c1e00640 1 -- 172.21.15.2:0/1069052964 wait complete. 2024-03-11T18:31:04.516 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:31:04.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 -- 172.21.15.2:0/562440535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468073b80 msgr2=0x7ff468073fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 --2- 172.21.15.2:0/562440535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468073b80 0x7ff468073fe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff450009ca0 tx=0x7ff45002c180 comp rx=0 tx=0).stop 2024-03-11T18:31:04.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 -- 172.21.15.2:0/562440535 shutdown_connections 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 --2- 172.21.15.2:0/562440535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468108f30 0x7ff46810b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 --2- 172.21.15.2:0/562440535 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468073b80 0x7ff468073fe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 --2- 172.21.15.2:0/562440535 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff4680735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.732+0000 7ff467200640 1 -- 172.21.15.2:0/562440535 >> 172.21.15.2:0/562440535 conn(0x7ff4680fc610 msgr2=0x7ff4680fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.733+0000 7ff467200640 1 -- 172.21.15.2:0/562440535 shutdown_connections 2024-03-11T18:31:04.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.733+0000 7ff467200640 1 -- 172.21.15.2:0/562440535 wait complete. 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.733+0000 7ff467200640 1 Processor -- start 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 -- start start 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468073b80 0x7ff4681992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468108f30 0x7ff4681a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a0dc0 con 0x7ff468075160 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a0f30 con 0x7ff468073b80 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff467200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4681a10f0 con 0x7ff468108f30 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff465400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff465400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39696/0 (socket says 172.21.15.2:39696) 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff465400640 1 -- 172.21.15.2:0/328589094 learned_addr learned my addr 172.21.15.2:0/328589094 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff465e00640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468073b80 0x7ff4681992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.734+0000 7ff466800640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468108f30 0x7ff4681a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 -- 172.21.15.2:0/328589094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468108f30 msgr2=0x7ff4681a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468108f30 0x7ff4681a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 -- 172.21.15.2:0/328589094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468073b80 msgr2=0x7ff4681992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468073b80 0x7ff4681992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 -- 172.21.15.2:0/328589094 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450009900 con 0x7ff468075160 2024-03-11T18:31:04.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff465400640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff4500326c0 tx=0x7ff450034ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450032890 con 0x7ff468075160 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.735+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4681a1370 con 0x7ff468075160 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.736+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff450004040 con 0x7ff468075160 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.736+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450032ed0 con 0x7ff468075160 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.736+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4681a1870 con 0x7ff468075160 2024-03-11T18:31:04.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.737+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff468107cf0 con 0x7ff468075160 2024-03-11T18:31:04.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.739+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7ff45000a040 con 0x7ff468075160 2024-03-11T18:31:04.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.740+0000 7ff45ea00640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 0x7ff4480a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:04.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.740+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff4500f0140 con 0x7ff468075160 2024-03-11T18:31:04.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.740+0000 7ff465e00640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 0x7ff4480a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:04.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.741+0000 7ff465e00640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 0x7ff4480a3ff0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff458005d00 tx=0x7ff45800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:04.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.744+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4500b9060 con 0x7ff468075160 2024-03-11T18:31:04.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:04 smithi002 ceph-mon[31328]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:04 smithi002 ceph-mon[31328]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:04 smithi165 ceph-mon[36634]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:04 smithi165 ceph-mon[36634]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:04 smithi023 ceph-mon[36406]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:04 smithi023 ceph-mon[36406]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:31:04.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.943+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff46810ef00 con 0x7ff468075160 2024-03-11T18:31:04.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.946+0000 7ff45ea00640 1 -- 172.21.15.2:0/328589094 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7ff4500bc720 con 0x7ff468075160 2024-03-11T18:31:04.948 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:04.948 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:31:04.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.948+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 msgr2=0x7ff4480a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.948+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 0x7ff4480a3ff0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff458005d00 tx=0x7ff45800b410 comp rx=0 tx=0).stop 2024-03-11T18:31:04.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.948+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 msgr2=0x7ff468199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:04.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.948+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff4500326c0 tx=0x7ff450034ea0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 shutdown_connections 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4480a1b30 0x7ff4480a3ff0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff468108f30 0x7ff4681a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff468075160 0x7ff468199810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 --2- 172.21.15.2:0/328589094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff468073b80 0x7ff4681992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 >> 172.21.15.2:0/328589094 conn(0x7ff4680fc610 msgr2=0x7ff468104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 shutdown_connections 2024-03-11T18:31:04.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:04.949+0000 7ff467200640 1 -- 172.21.15.2:0/328589094 wait complete. 2024-03-11T18:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-11T18:31:04.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 -- 172.21.15.2:0/802963162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 msgr2=0x7fc73c073fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 --2- 172.21.15.2:0/802963162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c073fe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc724009900 tx=0x7fc72402bcc0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 -- 172.21.15.2:0/802963162 shutdown_connections 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 --2- 172.21.15.2:0/802963162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c108f30 0x7fc73c10b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 --2- 172.21.15.2:0/802963162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c073fe0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 --2- 172.21.15.2:0/802963162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c075160 0x7fc73c0735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 -- 172.21.15.2:0/802963162 >> 172.21.15.2:0/802963162 conn(0x7fc73c0fc610 msgr2=0x7fc73c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 -- 172.21.15.2:0/802963162 shutdown_connections 2024-03-11T18:31:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.194+0000 7fc741800640 1 -- 172.21.15.2:0/802963162 wait complete. 2024-03-11T18:31:05.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 Processor -- start 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 -- start start 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c108f30 0x7fc73c1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c1a0e10 con 0x7fc73c073b80 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c1a0f80 con 0x7fc73c075160 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.195+0000 7fc741800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c1a1140 con 0x7fc73c108f30 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc73be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47456/0 (socket says 172.21.15.2:47456) 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 -- 172.21.15.2:0/3005774515 learned_addr learned my addr 172.21.15.2:0/3005774515 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc740e00640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c108f30 0x7fc73c1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 -- 172.21.15.2:0/3005774515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c108f30 msgr2=0x7fc73c1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c108f30 0x7fc73c1a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 -- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 msgr2=0x7fc73c199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 -- 172.21.15.2:0/3005774515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009560 con 0x7fc73c075160 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.196+0000 7fc733400640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc724006010 tx=0x7fc724005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.197+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7240043d0 con 0x7fc73c075160 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.197+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7240052d0 con 0x7fc73c075160 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.197+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c1a13c0 con 0x7fc73c075160 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.197+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc724003cc0 con 0x7fc73c075160 2024-03-11T18:31:05.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.197+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c1a18d0 con 0x7fc73c075160 2024-03-11T18:31:05.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.198+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c107cf0 con 0x7fc73c075160 2024-03-11T18:31:05.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.200+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fc724005440 con 0x7fc73c075160 2024-03-11T18:31:05.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.201+0000 7fc739600640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 0x7fc7200a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.201+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc7240f02d0 con 0x7fc73c075160 2024-03-11T18:31:05.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.202+0000 7fc73be00640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 0x7fc7200a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.202+0000 7fc73be00640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 0x7fc7200a3f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009bd0 tx=0x7fc72c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:05.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.204+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc7240b91f0 con 0x7fc73c075160 2024-03-11T18:31:05.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.402+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc73c0615c0 con 0x7fc7200a1ab0 2024-03-11T18:31:05.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.415+0000 7fc739600640 1 -- 172.21.15.2:0/3005774515 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340358 (secure 0 0 0) 0x7fc73c0615c0 con 0x7fc7200a1ab0 2024-03-11T18:31:05.423 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:05.425 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.422+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 msgr2=0x7fc7200a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.422+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 0x7fc7200a3f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009bd0 tx=0x7fc72c005f90 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 msgr2=0x7fc73c199860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc724006010 tx=0x7fc724005e80 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 shutdown_connections 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7200a1ab0 0x7fc7200a3f70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc73c108f30 0x7fc73c1a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc73c075160 0x7fc73c199860 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 --2- 172.21.15.2:0/3005774515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc73c073b80 0x7fc73c199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 >> 172.21.15.2:0/3005774515 conn(0x7fc73c0fc610 msgr2=0x7fc73c104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:05.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 shutdown_connections 2024-03-11T18:31:05.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.423+0000 7fc741800640 1 -- 172.21.15.2:0/3005774515 wait complete. 2024-03-11T18:31:05.443 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-11T18:31:04.211195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598328,"num_objects":61,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":151,"num_read_kb":132,"num_write":308,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366976,"kb_used_data":13660,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574848,"statfs":{"total":1151940427776,"available":1151564644352,"internally_reserved":0,"allocated":13987840,"data_stored":7598916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3461,"num_objects":37,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":15,"num_read_kb":13,"num_write":79,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006628"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057433+0000","last_change":"2024-03-11T18:30:28.060909+0000","last_active":"2024-03-11T18:30:34.057433+0000","last_peered":"2024-03-11T18:30:34.057433+0000","last_clean":"2024-03-11T18:30:34.057433+0000","last_became_active":"2024-03-11T18:30:28.060554+0000","last_became_peered":"2024-03-11T18:30:28.060554+0000","last_unstale":"2024-03-11T18:30:34.057433+0000","last_undegraded":"2024-03-11T18:30:34.057433+0000","last_fullsized":"2024-03-11T18:30:34.057433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:33.609820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844312+0000","last_change":"2024-03-11T18:30:28.055136+0000","last_active":"2024-03-11T18:30:32.844312+0000","last_peered":"2024-03-11T18:30:32.844312+0000","last_clean":"2024-03-11T18:30:32.844312+0000","last_became_active":"2024-03-11T18:30:28.054785+0000","last_became_peered":"2024-03-11T18:30:28.054785+0000","last_unstale":"2024-03-11T18:30:32.844312+0000","last_undegraded":"2024-03-11T18:30:32.844312+0000","last_fullsized":"2024-03-11T18:30:32.844312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:12.702850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843458+0000","last_change":"2024-03-11T18:30:28.051593+0000","last_active":"2024-03-11T18:30:32.843458+0000","last_peered":"2024-03-11T18:30:32.843458+0000","last_clean":"2024-03-11T18:30:32.843458+0000","last_became_active":"2024-03-11T18:30:28.049344+0000","last_became_peered":"2024-03-11T18:30:28.049344+0000","last_unstale":"2024-03-11T18:30:32.843458+0000","last_undegraded":"2024-03-11T18:30:32.843458+0000","last_fullsized":"2024-03-11T18:30:32.843458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:06.433202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845213+0000","last_change":"2024-03-11T18:30:28.036335+0000","last_active":"2024-03-11T18:30:32.845213+0000","last_peered":"2024-03-11T18:30:32.845213+0000","last_clean":"2024-03-11T18:30:32.845213+0000","last_became_active":"2024-03-11T18:30:28.035728+0000","last_became_peered":"2024-03-11T18:30:28.035728+0000","last_unstale":"2024-03-11T18:30:32.845213+0000","last_undegraded":"2024-03-11T18:30:32.845213+0000","last_fullsized":"2024-03-11T18:30:32.845213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:10.728266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055316+0000","last_change":"2024-03-11T18:30:28.050560+0000","last_active":"2024-03-11T18:30:34.055316+0000","last_peered":"2024-03-11T18:30:34.055316+0000","last_clean":"2024-03-11T18:30:34.055316+0000","last_became_active":"2024-03-11T18:30:28.050273+0000","last_became_peered":"2024-03-11T18:30:28.050273+0000","last_unstale":"2024-03-11T18:30:34.055316+0000","last_undegraded":"2024-03-11T18:30:34.055316+0000","last_fullsized":"2024-03-11T18:30:34.055316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:56.780575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432062+0000","last_change":"2024-03-11T18:30:28.056879+0000","last_active":"2024-03-11T18:30:33.432062+0000","last_peered":"2024-03-11T18:30:33.432062+0000","last_clean":"2024-03-11T18:30:33.432062+0000","last_became_active":"2024-03-11T18:30:28.056435+0000","last_became_peered":"2024-03-11T18:30:28.056435+0000","last_unstale":"2024-03-11T18:30:33.432062+0000","last_undegraded":"2024-03-11T18:30:33.432062+0000","last_fullsized":"2024-03-11T18:30:33.432062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:04.595503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962340+0000","last_change":"2024-03-11T18:30:28.044355+0000","last_active":"2024-03-11T18:30:33.962340+0000","last_peered":"2024-03-11T18:30:33.962340+0000","last_clean":"2024-03-11T18:30:33.962340+0000","last_became_active":"2024-03-11T18:30:28.043875+0000","last_became_peered":"2024-03-11T18:30:28.043875+0000","last_unstale":"2024-03-11T18:30:33.962340+0000","last_undegraded":"2024-03-11T18:30:33.962340+0000","last_fullsized":"2024-03-11T18:30:33.962340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:25.924359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059316+0000","last_change":"2024-03-11T18:30:28.055633+0000","last_active":"2024-03-11T18:30:34.059316+0000","last_peered":"2024-03-11T18:30:34.059316+0000","last_clean":"2024-03-11T18:30:34.059316+0000","last_became_active":"2024-03-11T18:30:28.055158+0000","last_became_peered":"2024-03-11T18:30:28.055158+0000","last_unstale":"2024-03-11T18:30:34.059316+0000","last_undegraded":"2024-03-11T18:30:34.059316+0000","last_fullsized":"2024-03-11T18:30:34.059316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:21:45.379551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055468+0000","last_change":"2024-03-11T18:30:28.037306+0000","last_active":"2024-03-11T18:30:34.055468+0000","last_peered":"2024-03-11T18:30:34.055468+0000","last_clean":"2024-03-11T18:30:34.055468+0000","last_became_active":"2024-03-11T18:30:28.036930+0000","last_became_peered":"2024-03-11T18:30:28.036930+0000","last_unstale":"2024-03-11T18:30:34.055468+0000","last_undegraded":"2024-03-11T18:30:34.055468+0000","last_fullsized":"2024-03-11T18:30:34.055468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432512+0000","last_change":"2024-03-11T18:30:28.038914+0000","last_active":"2024-03-11T18:30:33.432512+0000","last_peered":"2024-03-11T18:30:33.432512+0000","last_clean":"2024-03-11T18:30:33.432512+0000","last_became_active":"2024-03-11T18:30:28.038266+0000","last_became_peered":"2024-03-11T18:30:28.038266+0000","last_unstale":"2024-03-11T18:30:33.432512+0000","last_undegraded":"2024-03-11T18:30:33.432512+0000","last_fullsized":"2024-03-11T18:30:33.432512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:59.136860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963101+0000","last_change":"2024-03-11T18:30:28.041833+0000","last_active":"2024-03-11T18:30:33.963101+0000","last_peered":"2024-03-11T18:30:33.963101+0000","last_clean":"2024-03-11T18:30:33.963101+0000","last_became_active":"2024-03-11T18:30:28.041307+0000","last_became_peered":"2024-03-11T18:30:28.041307+0000","last_unstale":"2024-03-11T18:30:33.963101+0000","last_undegraded":"2024-03-11T18:30:33.963101+0000","last_fullsized":"2024-03-11T18:30:33.963101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:35.803202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845101+0000","last_change":"2024-03-11T18:30:28.037646+0000","last_active":"2024-03-11T18:30:32.845101+0000","last_peered":"2024-03-11T18:30:32.845101+0000","last_clean":"2024-03-11T18:30:32.845101+0000","last_became_active":"2024-03-11T18:30:28.037272+0000","last_became_peered":"2024-03-11T18:30:28.037272+0000","last_unstale":"2024-03-11T18:30:32.845101+0000","last_undegraded":"2024-03-11T18:30:32.845101+0000","last_fullsized":"2024-03-11T18:30:32.845101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:50.594666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060405+0000","last_change":"2024-03-11T18:30:28.056917+0000","last_active":"2024-03-11T18:30:34.060405+0000","last_peered":"2024-03-11T18:30:34.060405+0000","last_clean":"2024-03-11T18:30:34.060405+0000","last_became_active":"2024-03-11T18:30:28.056496+0000","last_became_peered":"2024-03-11T18:30:28.056496+0000","last_unstale":"2024-03-11T18:30:34.060405+0000","last_undegraded":"2024-03-11T18:30:34.060405+0000","last_fullsized":"2024-03-11T18:30:34.060405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:44.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842819+0000","last_change":"2024-03-11T18:30:28.055180+0000","last_active":"2024-03-11T18:30:32.842819+0000","last_peered":"2024-03-11T18:30:32.842819+0000","last_clean":"2024-03-11T18:30:32.842819+0000","last_became_active":"2024-03-11T18:30:28.054807+0000","last_became_peered":"2024-03-11T18:30:28.054807+0000","last_unstale":"2024-03-11T18:30:32.842819+0000","last_undegraded":"2024-03-11T18:30:32.842819+0000","last_fullsized":"2024-03-11T18:30:32.842819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:31.671461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842796+0000","last_change":"2024-03-11T18:30:28.065015+0000","last_active":"2024-03-11T18:30:32.842796+0000","last_peered":"2024-03-11T18:30:32.842796+0000","last_clean":"2024-03-11T18:30:32.842796+0000","last_became_active":"2024-03-11T18:30:28.061131+0000","last_became_peered":"2024-03-11T18:30:28.061131+0000","last_unstale":"2024-03-11T18:30:32.842796+0000","last_undegraded":"2024-03-11T18:30:32.842796+0000","last_fullsized":"2024-03-11T18:30:32.842796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:35.314562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.054769+0000","last_change":"2024-03-11T18:30:28.054388+0000","last_active":"2024-03-11T18:30:34.054769+0000","last_peered":"2024-03-11T18:30:34.054769+0000","last_clean":"2024-03-11T18:30:34.054769+0000","last_became_active":"2024-03-11T18:30:28.054036+0000","last_became_peered":"2024-03-11T18:30:28.054036+0000","last_unstale":"2024-03-11T18:30:34.054769+0000","last_undegraded":"2024-03-11T18:30:34.054769+0000","last_fullsized":"2024-03-11T18:30:34.054769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:45:21.476944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057326+0000","last_change":"2024-03-11T18:30:28.061935+0000","last_active":"2024-03-11T18:30:34.057326+0000","last_peered":"2024-03-11T18:30:34.057326+0000","last_clean":"2024-03-11T18:30:34.057326+0000","last_became_active":"2024-03-11T18:30:28.061558+0000","last_became_peered":"2024-03-11T18:30:28.061558+0000","last_unstale":"2024-03-11T18:30:34.057326+0000","last_undegraded":"2024-03-11T18:30:34.057326+0000","last_fullsized":"2024-03-11T18:30:34.057326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:01.728956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058341+0000","last_change":"2024-03-11T18:30:28.058372+0000","last_active":"2024-03-11T18:30:34.058341+0000","last_peered":"2024-03-11T18:30:34.058341+0000","last_clean":"2024-03-11T18:30:34.058341+0000","last_became_active":"2024-03-11T18:30:28.058028+0000","last_became_peered":"2024-03-11T18:30:28.058028+0000","last_unstale":"2024-03-11T18:30:34.058341+0000","last_undegraded":"2024-03-11T18:30:34.058341+0000","last_fullsized":"2024-03-11T18:30:34.058341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:20:15.645368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842261+0000","last_change":"2024-03-11T18:30:28.054845+0000","last_active":"2024-03-11T18:30:32.842261+0000","last_peered":"2024-03-11T18:30:32.842261+0000","last_clean":"2024-03-11T18:30:32.842261+0000","last_became_active":"2024-03-11T18:30:28.054497+0000","last_became_peered":"2024-03-11T18:30:28.054497+0000","last_unstale":"2024-03-11T18:30:32.842261+0000","last_undegraded":"2024-03-11T18:30:32.842261+0000","last_fullsized":"2024-03-11T18:30:32.842261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:52.100045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842262+0000","last_change":"2024-03-11T18:30:32.322103+0000","last_active":"2024-03-11T18:30:32.842262+0000","last_peered":"2024-03-11T18:30:32.842262+0000","last_clean":"2024-03-11T18:30:32.842262+0000","last_became_active":"2024-03-11T18:30:32.321422+0000","last_became_peered":"2024-03-11T18:30:32.321422+0000","last_unstale":"2024-03-11T18:30:32.842262+0000","last_undegraded":"2024-03-11T18:30:32.842262+0000","last_fullsized":"2024-03-11T18:30:32.842262+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:56.946318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.464119+0000","last_change":"2024-03-11T18:30:27.283463+0000","last_active":"2024-03-11T18:31:00.464119+0000","last_peered":"2024-03-11T18:31:00.464119+0000","last_clean":"2024-03-11T18:31:00.464119+0000","last_became_active":"2024-03-11T18:30:27.279100+0000","last_became_peered":"2024-03-11T18:30:27.279100+0000","last_unstale":"2024-03-11T18:31:00.464119+0000","last_undegraded":"2024-03-11T18:31:00.464119+0000","last_fullsized":"2024-03-11T18:31:00.464119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:38.096395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063150+0000","last_change":"2024-03-11T18:30:28.051288+0000","last_active":"2024-03-11T18:30:34.063150+0000","last_peered":"2024-03-11T18:30:34.063150+0000","last_clean":"2024-03-11T18:30:34.063150+0000","last_became_active":"2024-03-11T18:30:28.050726+0000","last_became_peered":"2024-03-11T18:30:28.050726+0000","last_unstale":"2024-03-11T18:30:34.063150+0000","last_undegraded":"2024-03-11T18:30:34.063150+0000","last_fullsized":"2024-03-11T18:30:34.063150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:29.125099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844657+0000","last_change":"2024-03-11T18:30:32.311188+0000","last_active":"2024-03-11T18:30:32.844657+0000","last_peered":"2024-03-11T18:30:32.844657+0000","last_clean":"2024-03-11T18:30:32.844657+0000","last_became_active":"2024-03-11T18:30:32.310670+0000","last_became_peered":"2024-03-11T18:30:32.310670+0000","last_unstale":"2024-03-11T18:30:32.844657+0000","last_undegraded":"2024-03-11T18:30:32.844657+0000","last_fullsized":"2024-03-11T18:30:32.844657+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:51:17.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063752+0000","last_change":"2024-03-11T18:30:27.284990+0000","last_active":"2024-03-11T18:30:34.063752+0000","last_peered":"2024-03-11T18:30:34.063752+0000","last_clean":"2024-03-11T18:30:34.063752+0000","last_became_active":"2024-03-11T18:30:27.283629+0000","last_became_peered":"2024-03-11T18:30:27.283629+0000","last_unstale":"2024-03-11T18:30:34.063752+0000","last_undegraded":"2024-03-11T18:30:34.063752+0000","last_fullsized":"2024-03-11T18:30:34.063752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:33.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844512+0000","last_change":"2024-03-11T18:30:28.066345+0000","last_active":"2024-03-11T18:30:32.844512+0000","last_peered":"2024-03-11T18:30:32.844512+0000","last_clean":"2024-03-11T18:30:32.844512+0000","last_became_active":"2024-03-11T18:30:28.061654+0000","last_became_peered":"2024-03-11T18:30:28.061654+0000","last_unstale":"2024-03-11T18:30:32.844512+0000","last_undegraded":"2024-03-11T18:30:32.844512+0000","last_fullsized":"2024-03-11T18:30:32.844512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:29.651510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844691+0000","last_change":"2024-03-11T18:30:32.318088+0000","last_active":"2024-03-11T18:30:32.844691+0000","last_peered":"2024-03-11T18:30:32.844691+0000","last_clean":"2024-03-11T18:30:32.844691+0000","last_became_active":"2024-03-11T18:30:32.317304+0000","last_became_peered":"2024-03-11T18:30:32.317304+0000","last_unstale":"2024-03-11T18:30:32.844691+0000","last_undegraded":"2024-03-11T18:30:32.844691+0000","last_fullsized":"2024-03-11T18:30:32.844691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:01.678144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.457547+0000","last_change":"2024-03-11T18:30:27.268370+0000","last_active":"2024-03-11T18:31:00.457547+0000","last_peered":"2024-03-11T18:31:00.457547+0000","last_clean":"2024-03-11T18:31:00.457547+0000","last_became_active":"2024-03-11T18:30:27.268033+0000","last_became_peered":"2024-03-11T18:30:27.268033+0000","last_unstale":"2024-03-11T18:31:00.457547+0000","last_undegraded":"2024-03-11T18:31:00.457547+0000","last_fullsized":"2024-03-11T18:31:00.457547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:31:13.545893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055113+0000","last_change":"2024-03-11T18:30:28.051611+0000","last_active":"2024-03-11T18:30:34.055113+0000","last_peered":"2024-03-11T18:30:34.055113+0000","last_clean":"2024-03-11T18:30:34.055113+0000","last_became_active":"2024-03-11T18:30:28.051249+0000","last_became_peered":"2024-03-11T18:30:28.051249+0000","last_unstale":"2024-03-11T18:30:34.055113+0000","last_undegraded":"2024-03-11T18:30:34.055113+0000","last_fullsized":"2024-03-11T18:30:34.055113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:12.942910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844496+0000","last_change":"2024-03-11T18:30:32.318934+0000","last_active":"2024-03-11T18:30:32.844496+0000","last_peered":"2024-03-11T18:30:32.844496+0000","last_clean":"2024-03-11T18:30:32.844496+0000","last_became_active":"2024-03-11T18:30:32.318474+0000","last_became_peered":"2024-03-11T18:30:32.318474+0000","last_unstale":"2024-03-11T18:30:32.844496+0000","last_undegraded":"2024-03-11T18:30:32.844496+0000","last_fullsized":"2024-03-11T18:30:32.844496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:48:33.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055702+0000","last_change":"2024-03-11T18:30:27.268675+0000","last_active":"2024-03-11T18:30:34.055702+0000","last_peered":"2024-03-11T18:30:34.055702+0000","last_clean":"2024-03-11T18:30:34.055702+0000","last_became_active":"2024-03-11T18:30:27.268320+0000","last_became_peered":"2024-03-11T18:30:27.268320+0000","last_unstale":"2024-03-11T18:30:34.055702+0000","last_undegraded":"2024-03-11T18:30:34.055702+0000","last_fullsized":"2024-03-11T18:30:34.055702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:06.439930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056278+0000","last_change":"2024-03-11T18:30:28.038380+0000","last_active":"2024-03-11T18:30:34.056278+0000","last_peered":"2024-03-11T18:30:34.056278+0000","last_clean":"2024-03-11T18:30:34.056278+0000","last_became_active":"2024-03-11T18:30:28.037878+0000","last_became_peered":"2024-03-11T18:30:28.037878+0000","last_unstale":"2024-03-11T18:30:34.056278+0000","last_undegraded":"2024-03-11T18:30:34.056278+0000","last_fullsized":"2024-03-11T18:30:34.056278+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:10.988295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842311+0000","last_change":"2024-03-11T18:30:32.304747+0000","last_active":"2024-03-11T18:30:32.842311+0000","last_peered":"2024-03-11T18:30:32.842311+0000","last_clean":"2024-03-11T18:30:32.842311+0000","last_became_active":"2024-03-11T18:30:32.304282+0000","last_became_peered":"2024-03-11T18:30:32.304282+0000","last_unstale":"2024-03-11T18:30:32.842311+0000","last_undegraded":"2024-03-11T18:30:32.842311+0000","last_fullsized":"2024-03-11T18:30:32.842311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:42.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432710+0000","last_change":"2024-03-11T18:30:27.267574+0000","last_active":"2024-03-11T18:30:59.432710+0000","last_peered":"2024-03-11T18:30:59.432710+0000","last_clean":"2024-03-11T18:30:59.432710+0000","last_became_active":"2024-03-11T18:30:27.267278+0000","last_became_peered":"2024-03-11T18:30:27.267278+0000","last_unstale":"2024-03-11T18:30:59.432710+0000","last_undegraded":"2024-03-11T18:30:59.432710+0000","last_fullsized":"2024-03-11T18:30:59.432710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:28.997557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055488+0000","last_change":"2024-03-11T18:30:28.060280+0000","last_active":"2024-03-11T18:30:34.055488+0000","last_peered":"2024-03-11T18:30:34.055488+0000","last_clean":"2024-03-11T18:30:34.055488+0000","last_became_active":"2024-03-11T18:30:28.059929+0000","last_became_peered":"2024-03-11T18:30:28.059929+0000","last_unstale":"2024-03-11T18:30:34.055488+0000","last_undegraded":"2024-03-11T18:30:34.055488+0000","last_fullsized":"2024-03-11T18:30:34.055488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:47:31.995021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842034+0000","last_change":"2024-03-11T18:30:32.321754+0000","last_active":"2024-03-11T18:30:32.842034+0000","last_peered":"2024-03-11T18:30:32.842034+0000","last_clean":"2024-03-11T18:30:32.842034+0000","last_became_active":"2024-03-11T18:30:32.321329+0000","last_became_peered":"2024-03-11T18:30:32.321329+0000","last_unstale":"2024-03-11T18:30:32.842034+0000","last_undegraded":"2024-03-11T18:30:32.842034+0000","last_fullsized":"2024-03-11T18:30:32.842034+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:48.985005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432007+0000","last_change":"2024-03-11T18:30:27.285442+0000","last_active":"2024-03-11T18:30:33.432007+0000","last_peered":"2024-03-11T18:30:33.432007+0000","last_clean":"2024-03-11T18:30:33.432007+0000","last_became_active":"2024-03-11T18:30:27.285127+0000","last_became_peered":"2024-03-11T18:30:27.285127+0000","last_unstale":"2024-03-11T18:30:33.432007+0000","last_undegraded":"2024-03-11T18:30:33.432007+0000","last_fullsized":"2024-03-11T18:30:33.432007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:01.278846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056379+0000","last_change":"2024-03-11T18:30:28.036105+0000","last_active":"2024-03-11T18:30:34.056379+0000","last_peered":"2024-03-11T18:30:34.056379+0000","last_clean":"2024-03-11T18:30:34.056379+0000","last_became_active":"2024-03-11T18:30:28.035713+0000","last_became_peered":"2024-03-11T18:30:28.035713+0000","last_unstale":"2024-03-11T18:30:34.056379+0000","last_undegraded":"2024-03-11T18:30:34.056379+0000","last_fullsized":"2024-03-11T18:30:34.056379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:59.363573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057419+0000","last_change":"2024-03-11T18:30:32.327126+0000","last_active":"2024-03-11T18:30:34.057419+0000","last_peered":"2024-03-11T18:30:34.057419+0000","last_clean":"2024-03-11T18:30:34.057419+0000","last_became_active":"2024-03-11T18:30:32.326668+0000","last_became_peered":"2024-03-11T18:30:32.326668+0000","last_unstale":"2024-03-11T18:30:34.057419+0000","last_undegraded":"2024-03-11T18:30:34.057419+0000","last_fullsized":"2024-03-11T18:30:34.057419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:03.231714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465167+0000","last_change":"2024-03-11T18:30:27.289439+0000","last_active":"2024-03-11T18:31:00.465167+0000","last_peered":"2024-03-11T18:31:00.465167+0000","last_clean":"2024-03-11T18:31:00.465167+0000","last_became_active":"2024-03-11T18:30:27.286067+0000","last_became_peered":"2024-03-11T18:30:27.286067+0000","last_unstale":"2024-03-11T18:31:00.465167+0000","last_undegraded":"2024-03-11T18:31:00.465167+0000","last_fullsized":"2024-03-11T18:31:00.465167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:35.522841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059494+0000","last_change":"2024-03-11T18:30:28.058525+0000","last_active":"2024-03-11T18:30:34.059494+0000","last_peered":"2024-03-11T18:30:34.059494+0000","last_clean":"2024-03-11T18:30:34.059494+0000","last_became_active":"2024-03-11T18:30:28.058122+0000","last_became_peered":"2024-03-11T18:30:28.058122+0000","last_unstale":"2024-03-11T18:30:34.059494+0000","last_undegraded":"2024-03-11T18:30:34.059494+0000","last_fullsized":"2024-03-11T18:30:34.059494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:45:04.537880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843998+0000","last_change":"2024-03-11T18:30:32.314757+0000","last_active":"2024-03-11T18:30:32.843998+0000","last_peered":"2024-03-11T18:30:32.843998+0000","last_clean":"2024-03-11T18:30:32.843998+0000","last_became_active":"2024-03-11T18:30:32.313461+0000","last_became_peered":"2024-03-11T18:30:32.313461+0000","last_unstale":"2024-03-11T18:30:32.843998+0000","last_undegraded":"2024-03-11T18:30:32.843998+0000","last_fullsized":"2024-03-11T18:30:32.843998+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:11:27.881788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055764+0000","last_change":"2024-03-11T18:30:27.268359+0000","last_active":"2024-03-11T18:30:34.055764+0000","last_peered":"2024-03-11T18:30:34.055764+0000","last_clean":"2024-03-11T18:30:34.055764+0000","last_became_active":"2024-03-11T18:30:27.268045+0000","last_became_peered":"2024-03-11T18:30:27.268045+0000","last_unstale":"2024-03-11T18:30:34.055764+0000","last_undegraded":"2024-03-11T18:30:34.055764+0000","last_fullsized":"2024-03-11T18:30:34.055764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:53.678477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056587+0000","last_change":"2024-03-11T18:30:28.051331+0000","last_active":"2024-03-11T18:30:34.056587+0000","last_peered":"2024-03-11T18:30:34.056587+0000","last_clean":"2024-03-11T18:30:34.056587+0000","last_became_active":"2024-03-11T18:30:28.050462+0000","last_became_peered":"2024-03-11T18:30:28.050462+0000","last_unstale":"2024-03-11T18:30:34.056587+0000","last_undegraded":"2024-03-11T18:30:34.056587+0000","last_fullsized":"2024-03-11T18:30:34.056587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:24.194650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060286+0000","last_change":"2024-03-11T18:30:32.317173+0000","last_active":"2024-03-11T18:30:34.060286+0000","last_peered":"2024-03-11T18:30:34.060286+0000","last_clean":"2024-03-11T18:30:34.060286+0000","last_became_active":"2024-03-11T18:30:32.316721+0000","last_became_peered":"2024-03-11T18:30:32.316721+0000","last_unstale":"2024-03-11T18:30:34.060286+0000","last_undegraded":"2024-03-11T18:30:34.060286+0000","last_fullsized":"2024-03-11T18:30:34.060286+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:43.665949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432951+0000","last_change":"2024-03-11T18:30:27.279586+0000","last_active":"2024-03-11T18:30:59.432951+0000","last_peered":"2024-03-11T18:30:59.432951+0000","last_clean":"2024-03-11T18:30:59.432951+0000","last_became_active":"2024-03-11T18:30:27.279262+0000","last_became_peered":"2024-03-11T18:30:27.279262+0000","last_unstale":"2024-03-11T18:30:59.432951+0000","last_undegraded":"2024-03-11T18:30:59.432951+0000","last_fullsized":"2024-03-11T18:30:59.432951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:02:33.570755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055465+0000","last_change":"2024-03-11T18:30:28.060636+0000","last_active":"2024-03-11T18:30:34.055465+0000","last_peered":"2024-03-11T18:30:34.055465+0000","last_clean":"2024-03-11T18:30:34.055465+0000","last_became_active":"2024-03-11T18:30:28.060282+0000","last_became_peered":"2024-03-11T18:30:28.060282+0000","last_unstale":"2024-03-11T18:30:34.055465+0000","last_undegraded":"2024-03-11T18:30:34.055465+0000","last_fullsized":"2024-03-11T18:30:34.055465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:56.618563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056345+0000","last_change":"2024-03-11T18:30:32.314680+0000","last_active":"2024-03-11T18:30:34.056345+0000","last_peered":"2024-03-11T18:30:34.056345+0000","last_clean":"2024-03-11T18:30:34.056345+0000","last_became_active":"2024-03-11T18:30:32.314102+0000","last_became_peered":"2024-03-11T18:30:32.314102+0000","last_unstale":"2024-03-11T18:30:34.056345+0000","last_undegraded":"2024-03-11T18:30:34.056345+0000","last_fullsized":"2024-03-11T18:30:34.056345+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:18:55.989746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.431275+0000","last_change":"2024-03-11T18:30:27.267028+0000","last_active":"2024-03-11T18:30:59.431275+0000","last_peered":"2024-03-11T18:30:59.431275+0000","last_clean":"2024-03-11T18:30:59.431275+0000","last_became_active":"2024-03-11T18:30:27.266712+0000","last_became_peered":"2024-03-11T18:30:27.266712+0000","last_unstale":"2024-03-11T18:30:59.431275+0000","last_undegraded":"2024-03-11T18:30:59.431275+0000","last_fullsized":"2024-03-11T18:30:59.431275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:35.978113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844365+0000","last_change":"2024-03-11T18:30:28.066607+0000","last_active":"2024-03-11T18:30:32.844365+0000","last_peered":"2024-03-11T18:30:32.844365+0000","last_clean":"2024-03-11T18:30:32.844365+0000","last_became_active":"2024-03-11T18:30:28.061181+0000","last_became_peered":"2024-03-11T18:30:28.061181+0000","last_unstale":"2024-03-11T18:30:32.844365+0000","last_undegraded":"2024-03-11T18:30:32.844365+0000","last_fullsized":"2024-03-11T18:30:32.844365+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:01.203632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844947+0000","last_change":"2024-03-11T18:30:32.317424+0000","last_active":"2024-03-11T18:30:32.844947+0000","last_peered":"2024-03-11T18:30:32.844947+0000","last_clean":"2024-03-11T18:30:32.844947+0000","last_became_active":"2024-03-11T18:30:32.316633+0000","last_became_peered":"2024-03-11T18:30:32.316633+0000","last_unstale":"2024-03-11T18:30:32.844947+0000","last_undegraded":"2024-03-11T18:30:32.844947+0000","last_fullsized":"2024-03-11T18:30:32.844947+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:28:47.017770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056241+0000","last_change":"2024-03-11T18:30:27.270147+0000","last_active":"2024-03-11T18:30:34.056241+0000","last_peered":"2024-03-11T18:30:34.056241+0000","last_clean":"2024-03-11T18:30:34.056241+0000","last_became_active":"2024-03-11T18:30:27.269791+0000","last_became_peered":"2024-03-11T18:30:27.269791+0000","last_unstale":"2024-03-11T18:30:34.056241+0000","last_undegraded":"2024-03-11T18:30:34.056241+0000","last_fullsized":"2024-03-11T18:30:34.056241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:22.007518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055186+0000","last_change":"2024-03-11T18:30:28.035357+0000","last_active":"2024-03-11T18:30:34.055186+0000","last_peered":"2024-03-11T18:30:34.055186+0000","last_clean":"2024-03-11T18:30:34.055186+0000","last_became_active":"2024-03-11T18:30:28.035005+0000","last_became_peered":"2024-03-11T18:30:28.035005+0000","last_unstale":"2024-03-11T18:30:34.055186+0000","last_undegraded":"2024-03-11T18:30:34.055186+0000","last_fullsized":"2024-03-11T18:30:34.055186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:21.450440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056750+0000","last_change":"2024-03-11T18:30:32.323472+0000","last_active":"2024-03-11T18:30:34.056750+0000","last_peered":"2024-03-11T18:30:34.056750+0000","last_clean":"2024-03-11T18:30:34.056750+0000","last_became_active":"2024-03-11T18:30:32.321590+0000","last_became_peered":"2024-03-11T18:30:32.321590+0000","last_unstale":"2024-03-11T18:30:34.056750+0000","last_undegraded":"2024-03-11T18:30:34.056750+0000","last_fullsized":"2024-03-11T18:30:34.056750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:12.329127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.430384+0000","last_change":"2024-03-11T18:30:27.267683+0000","last_active":"2024-03-11T18:30:59.430384+0000","last_peered":"2024-03-11T18:30:59.430384+0000","last_clean":"2024-03-11T18:30:59.430384+0000","last_became_active":"2024-03-11T18:30:27.267333+0000","last_became_peered":"2024-03-11T18:30:27.267333+0000","last_unstale":"2024-03-11T18:30:59.430384+0000","last_undegraded":"2024-03-11T18:30:59.430384+0000","last_fullsized":"2024-03-11T18:30:59.430384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:33:18.721843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056403+0000","last_change":"2024-03-11T18:30:28.038846+0000","last_active":"2024-03-11T18:30:34.056403+0000","last_peered":"2024-03-11T18:30:34.056403+0000","last_clean":"2024-03-11T18:30:34.056403+0000","last_became_active":"2024-03-11T18:30:28.038479+0000","last_became_peered":"2024-03-11T18:30:28.038479+0000","last_unstale":"2024-03-11T18:30:34.056403+0000","last_undegraded":"2024-03-11T18:30:34.056403+0000","last_fullsized":"2024-03-11T18:30:34.056403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:25.877627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056767+0000","last_change":"2024-03-11T18:30:32.317761+0000","last_active":"2024-03-11T18:30:34.056767+0000","last_peered":"2024-03-11T18:30:34.056767+0000","last_clean":"2024-03-11T18:30:34.056767+0000","last_became_active":"2024-03-11T18:30:32.317438+0000","last_became_peered":"2024-03-11T18:30:32.317438+0000","last_unstale":"2024-03-11T18:30:34.056767+0000","last_undegraded":"2024-03-11T18:30:34.056767+0000","last_fullsized":"2024-03-11T18:30:34.056767+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:58.145037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.456980+0000","last_change":"2024-03-11T18:30:27.276641+0000","last_active":"2024-03-11T18:31:00.456980+0000","last_peered":"2024-03-11T18:31:00.456980+0000","last_clean":"2024-03-11T18:31:00.456980+0000","last_became_active":"2024-03-11T18:30:27.276293+0000","last_became_peered":"2024-03-11T18:30:27.276293+0000","last_unstale":"2024-03-11T18:31:00.456980+0000","last_undegraded":"2024-03-11T18:31:00.456980+0000","last_fullsized":"2024-03-11T18:31:00.456980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:46.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059448+0000","last_change":"2024-03-11T18:30:28.064304+0000","last_active":"2024-03-11T18:30:34.059448+0000","last_peered":"2024-03-11T18:30:34.059448+0000","last_clean":"2024-03-11T18:30:34.059448+0000","last_became_active":"2024-03-11T18:30:28.063923+0000","last_became_peered":"2024-03-11T18:30:28.063923+0000","last_unstale":"2024-03-11T18:30:34.059448+0000","last_undegraded":"2024-03-11T18:30:34.059448+0000","last_fullsized":"2024-03-11T18:30:34.059448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:57:41.289717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057170+0000","last_change":"2024-03-11T18:30:32.325409+0000","last_active":"2024-03-11T18:30:34.057170+0000","last_peered":"2024-03-11T18:30:34.057170+0000","last_clean":"2024-03-11T18:30:34.057170+0000","last_became_active":"2024-03-11T18:30:32.324953+0000","last_became_peered":"2024-03-11T18:30:32.324953+0000","last_unstale":"2024-03-11T18:30:34.057170+0000","last_undegraded":"2024-03-11T18:30:34.057170+0000","last_fullsized":"2024-03-11T18:30:34.057170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:44:54.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.434395+0000","last_change":"2024-03-11T18:30:27.271690+0000","last_active":"2024-03-11T18:30:59.434395+0000","last_peered":"2024-03-11T18:30:59.434395+0000","last_clean":"2024-03-11T18:30:59.434395+0000","last_became_active":"2024-03-11T18:30:27.271429+0000","last_became_peered":"2024-03-11T18:30:27.271429+0000","last_unstale":"2024-03-11T18:30:59.434395+0000","last_undegraded":"2024-03-11T18:30:59.434395+0000","last_fullsized":"2024-03-11T18:30:59.434395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:55:23.896642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962826+0000","last_change":"2024-03-11T18:30:28.054214+0000","last_active":"2024-03-11T18:30:33.962826+0000","last_peered":"2024-03-11T18:30:33.962826+0000","last_clean":"2024-03-11T18:30:33.962826+0000","last_became_active":"2024-03-11T18:30:28.053447+0000","last_became_peered":"2024-03-11T18:30:28.053447+0000","last_unstale":"2024-03-11T18:30:33.962826+0000","last_undegraded":"2024-03-11T18:30:33.962826+0000","last_fullsized":"2024-03-11T18:30:33.962826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:08:49.866509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058378+0000","last_change":"2024-03-11T18:30:32.325028+0000","last_active":"2024-03-11T18:30:34.058378+0000","last_peered":"2024-03-11T18:30:34.058378+0000","last_clean":"2024-03-11T18:30:34.058378+0000","last_became_active":"2024-03-11T18:30:32.324526+0000","last_became_peered":"2024-03-11T18:30:32.324526+0000","last_unstale":"2024-03-11T18:30:34.058378+0000","last_undegraded":"2024-03-11T18:30:34.058378+0000","last_fullsized":"2024-03-11T18:30:34.058378+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:08.031224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:02.473089+0000","last_change":"2024-03-11T18:30:27.265284+0000","last_active":"2024-03-11T18:31:02.473089+0000","last_peered":"2024-03-11T18:31:02.473089+0000","last_clean":"2024-03-11T18:31:02.473089+0000","last_became_active":"2024-03-11T18:30:27.264402+0000","last_became_peered":"2024-03-11T18:30:27.264402+0000","last_unstale":"2024-03-11T18:31:02.473089+0000","last_undegraded":"2024-03-11T18:31:02.473089+0000","last_fullsized":"2024-03-11T18:31:02.473089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:14.768956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962294+0000","last_change":"2024-03-11T18:30:28.040540+0000","last_active":"2024-03-11T18:30:33.962294+0000","last_peered":"2024-03-11T18:30:33.962294+0000","last_clean":"2024-03-11T18:30:33.962294+0000","last_became_active":"2024-03-11T18:30:28.040181+0000","last_became_peered":"2024-03-11T18:30:28.040181+0000","last_unstale":"2024-03-11T18:30:33.962294+0000","last_undegraded":"2024-03-11T18:30:33.962294+0000","last_fullsized":"2024-03-11T18:30:33.962294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:19.079250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061243+0000","last_change":"2024-03-11T18:30:32.326182+0000","last_active":"2024-03-11T18:30:34.061243+0000","last_peered":"2024-03-11T18:30:34.061243+0000","last_clean":"2024-03-11T18:30:34.061243+0000","last_became_active":"2024-03-11T18:30:32.325793+0000","last_became_peered":"2024-03-11T18:30:32.325793+0000","last_unstale":"2024-03-11T18:30:34.061243+0000","last_undegraded":"2024-03-11T18:30:34.061243+0000","last_fullsized":"2024-03-11T18:30:34.061243+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:14:13.212511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465352+0000","last_change":"2024-03-11T18:30:27.266724+0000","last_active":"2024-03-11T18:31:00.465352+0000","last_peered":"2024-03-11T18:31:00.465352+0000","last_clean":"2024-03-11T18:31:00.465352+0000","last_became_active":"2024-03-11T18:30:27.266394+0000","last_became_peered":"2024-03-11T18:30:27.266394+0000","last_unstale":"2024-03-11T18:31:00.465352+0000","last_undegraded":"2024-03-11T18:31:00.465352+0000","last_fullsized":"2024-03-11T18:31:00.465352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:24.115722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056087+0000","last_change":"2024-03-11T18:30:28.039476+0000","last_active":"2024-03-11T18:30:34.056087+0000","last_peered":"2024-03-11T18:30:34.056087+0000","last_clean":"2024-03-11T18:30:34.056087+0000","last_became_active":"2024-03-11T18:30:28.038897+0000","last_became_peered":"2024-03-11T18:30:28.038897+0000","last_unstale":"2024-03-11T18:30:34.056087+0000","last_undegraded":"2024-03-11T18:30:34.056087+0000","last_fullsized":"2024-03-11T18:30:34.056087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:36:16.103173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056012+0000","last_change":"2024-03-11T18:30:32.319516+0000","last_active":"2024-03-11T18:30:34.056012+0000","last_peered":"2024-03-11T18:30:34.056012+0000","last_clean":"2024-03-11T18:30:34.056012+0000","last_became_active":"2024-03-11T18:30:32.319117+0000","last_became_peered":"2024-03-11T18:30:32.319117+0000","last_unstale":"2024-03-11T18:30:34.056012+0000","last_undegraded":"2024-03-11T18:30:34.056012+0000","last_fullsized":"2024-03-11T18:30:34.056012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:39.797872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:01.453787+0000","last_change":"2024-03-11T18:30:27.267543+0000","last_active":"2024-03-11T18:31:01.453787+0000","last_peered":"2024-03-11T18:31:01.453787+0000","last_clean":"2024-03-11T18:31:01.453787+0000","last_became_active":"2024-03-11T18:30:27.267235+0000","last_became_peered":"2024-03-11T18:30:27.267235+0000","last_unstale":"2024-03-11T18:31:01.453787+0000","last_undegraded":"2024-03-11T18:31:01.453787+0000","last_fullsized":"2024-03-11T18:31:01.453787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:24.556896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057284+0000","last_change":"2024-03-11T18:30:28.039401+0000","last_active":"2024-03-11T18:30:34.057284+0000","last_peered":"2024-03-11T18:30:34.057284+0000","last_clean":"2024-03-11T18:30:34.057284+0000","last_became_active":"2024-03-11T18:30:28.039054+0000","last_became_peered":"2024-03-11T18:30:28.039054+0000","last_unstale":"2024-03-11T18:30:34.057284+0000","last_undegraded":"2024-03-11T18:30:34.057284+0000","last_fullsized":"2024-03-11T18:30:34.057284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:01.701304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055532+0000","last_change":"2024-03-11T18:30:32.527930+0000","last_active":"2024-03-11T18:30:34.055532+0000","last_peered":"2024-03-11T18:30:34.055532+0000","last_clean":"2024-03-11T18:30:34.055532+0000","last_became_active":"2024-03-11T18:30:32.527371+0000","last_became_peered":"2024-03-11T18:30:32.527371+0000","last_unstale":"2024-03-11T18:30:34.055532+0000","last_undegraded":"2024-03-11T18:30:34.055532+0000","last_fullsized":"2024-03-11T18:30:34.055532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:09:52.361924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057270+0000","last_change":"2024-03-11T18:30:27.281098+0000","last_active":"2024-03-11T18:30:34.057270+0000","last_peered":"2024-03-11T18:30:34.057270+0000","last_clean":"2024-03-11T18:30:34.057270+0000","last_became_active":"2024-03-11T18:30:27.280796+0000","last_became_peered":"2024-03-11T18:30:27.280796+0000","last_unstale":"2024-03-11T18:30:34.057270+0000","last_undegraded":"2024-03-11T18:30:34.057270+0000","last_fullsized":"2024-03-11T18:30:34.057270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:28.661626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432320+0000","last_change":"2024-03-11T18:30:28.054079+0000","last_active":"2024-03-11T18:30:33.432320+0000","last_peered":"2024-03-11T18:30:33.432320+0000","last_clean":"2024-03-11T18:30:33.432320+0000","last_became_active":"2024-03-11T18:30:28.053715+0000","last_became_peered":"2024-03-11T18:30:28.053715+0000","last_unstale":"2024-03-11T18:30:33.432320+0000","last_undegraded":"2024-03-11T18:30:33.432320+0000","last_fullsized":"2024-03-11T18:30:33.432320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:05.211602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.064381+0000","last_change":"2024-03-11T18:30:32.311132+0000","last_active":"2024-03-11T18:30:34.064381+0000","last_peered":"2024-03-11T18:30:34.064381+0000","last_clean":"2024-03-11T18:30:34.064381+0000","last_became_active":"2024-03-11T18:30:32.309585+0000","last_became_peered":"2024-03-11T18:30:32.309585+0000","last_unstale":"2024-03-11T18:30:34.064381+0000","last_undegraded":"2024-03-11T18:30:34.064381+0000","last_fullsized":"2024-03-11T18:30:34.064381+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:57.272479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962723+0000","last_change":"2024-03-11T18:30:27.288080+0000","last_active":"2024-03-11T18:30:33.962723+0000","last_peered":"2024-03-11T18:30:33.962723+0000","last_clean":"2024-03-11T18:30:33.962723+0000","last_became_active":"2024-03-11T18:30:27.287776+0000","last_became_peered":"2024-03-11T18:30:27.287776+0000","last_unstale":"2024-03-11T18:30:33.962723+0000","last_undegraded":"2024-03-11T18:30:33.962723+0000","last_fullsized":"2024-03-11T18:30:33.962723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:22.081753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432210+0000","last_change":"2024-03-11T18:30:28.062011+0000","last_active":"2024-03-11T18:30:33.432210+0000","last_peered":"2024-03-11T18:30:33.432210+0000","last_clean":"2024-03-11T18:30:33.432210+0000","last_became_active":"2024-03-11T18:30:28.061592+0000","last_became_peered":"2024-03-11T18:30:28.061592+0000","last_unstale":"2024-03-11T18:30:33.432210+0000","last_undegraded":"2024-03-11T18:30:33.432210+0000","last_fullsized":"2024-03-11T18:30:33.432210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:10.013980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061225+0000","last_change":"2024-03-11T18:30:32.528123+0000","last_active":"2024-03-11T18:30:34.061225+0000","last_peered":"2024-03-11T18:30:34.061225+0000","last_clean":"2024-03-11T18:30:34.061225+0000","last_became_active":"2024-03-11T18:30:32.527611+0000","last_became_peered":"2024-03-11T18:30:32.527611+0000","last_unstale":"2024-03-11T18:30:34.061225+0000","last_undegraded":"2024-03-11T18:30:34.061225+0000","last_fullsized":"2024-03-11T18:30:34.061225+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:28.501667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055953+0000","last_change":"2024-03-11T18:30:28.036930+0000","last_active":"2024-03-11T18:30:34.055953+0000","last_peered":"2024-03-11T18:30:34.055953+0000","last_clean":"2024-03-11T18:30:34.055953+0000","last_became_active":"2024-03-11T18:30:28.036286+0000","last_became_peered":"2024-03-11T18:30:28.036286+0000","last_unstale":"2024-03-11T18:30:34.055953+0000","last_undegraded":"2024-03-11T18:30:34.055953+0000","last_fullsized":"2024-03-11T18:30:34.055953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:10:40.239284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.468123+0000","last_change":"2024-03-11T18:30:27.287838+0000","last_active":"2024-03-11T18:31:00.468123+0000","last_peered":"2024-03-11T18:31:00.468123+0000","last_clean":"2024-03-11T18:31:00.468123+0000","last_became_active":"2024-03-11T18:30:27.287447+0000","last_became_peered":"2024-03-11T18:30:27.287447+0000","last_unstale":"2024-03-11T18:31:00.468123+0000","last_undegraded":"2024-03-11T18:31:00.468123+0000","last_fullsized":"2024-03-11T18:31:00.468123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:47.242977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963057+0000","last_change":"2024-03-11T18:30:32.321871+0000","last_active":"2024-03-11T18:30:33.963057+0000","last_peered":"2024-03-11T18:30:33.963057+0000","last_clean":"2024-03-11T18:30:33.963057+0000","last_became_active":"2024-03-11T18:30:32.321447+0000","last_became_peered":"2024-03-11T18:30:32.321447+0000","last_unstale":"2024-03-11T18:30:33.963057+0000","last_undegraded":"2024-03-11T18:30:33.963057+0000","last_fullsized":"2024-03-11T18:30:33.963057+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:56.124415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060833+0000","last_change":"2024-03-11T18:30:28.037906+0000","last_active":"2024-03-11T18:30:34.060833+0000","last_peered":"2024-03-11T18:30:34.060833+0000","last_clean":"2024-03-11T18:30:34.060833+0000","last_became_active":"2024-03-11T18:30:28.034143+0000","last_became_peered":"2024-03-11T18:30:28.034143+0000","last_unstale":"2024-03-11T18:30:34.060833+0000","last_undegraded":"2024-03-11T18:30:34.060833+0000","last_fullsized":"2024-03-11T18:30:34.060833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:36.728143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:01.463425+0000","last_change":"2024-03-11T18:30:27.273520+0000","last_active":"2024-03-11T18:31:01.463425+0000","last_peered":"2024-03-11T18:31:01.463425+0000","last_clean":"2024-03-11T18:31:01.463425+0000","last_became_active":"2024-03-11T18:30:27.273141+0000","last_became_peered":"2024-03-11T18:30:27.273141+0000","last_unstale":"2024-03-11T18:31:01.463425+0000","last_undegraded":"2024-03-11T18:31:01.463425+0000","last_fullsized":"2024-03-11T18:31:01.463425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:51.467579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962253+0000","last_change":"2024-03-11T18:29:08.539862+0000","last_active":"2024-03-11T18:30:33.962253+0000","last_peered":"2024-03-11T18:30:33.962253+0000","last_clean":"2024-03-11T18:30:33.962253+0000","last_became_active":"2024-03-11T18:29:08.539403+0000","last_became_peered":"2024-03-11T18:29:08.539403+0000","last_unstale":"2024-03-11T18:30:33.962253+0000","last_undegraded":"2024-03-11T18:30:33.962253+0000","last_fullsized":"2024-03-11T18:30:33.962253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_clean_scrub_stamp":"2024-03-11T18:25:13.035017+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:24.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057109+0000","last_change":"2024-03-11T18:30:32.528841+0000","last_active":"2024-03-11T18:30:34.057109+0000","last_peered":"2024-03-11T18:30:34.057109+0000","last_clean":"2024-03-11T18:30:34.057109+0000","last_became_active":"2024-03-11T18:30:32.528320+0000","last_became_peered":"2024-03-11T18:30:32.528320+0000","last_unstale":"2024-03-11T18:30:34.057109+0000","last_undegraded":"2024-03-11T18:30:34.057109+0000","last_fullsized":"2024-03-11T18:30:34.057109+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:26.354490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058730+0000","last_change":"2024-03-11T18:30:28.047552+0000","last_active":"2024-03-11T18:30:34.058730+0000","last_peered":"2024-03-11T18:30:34.058730+0000","last_clean":"2024-03-11T18:30:34.058730+0000","last_became_active":"2024-03-11T18:30:28.046470+0000","last_became_peered":"2024-03-11T18:30:28.046470+0000","last_unstale":"2024-03-11T18:30:34.058730+0000","last_undegraded":"2024-03-11T18:30:34.058730+0000","last_fullsized":"2024-03-11T18:30:34.058730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:58:23.080406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.458046+0000","last_change":"2024-03-11T18:30:27.269416+0000","last_active":"2024-03-11T18:31:00.458046+0000","last_peered":"2024-03-11T18:31:00.458046+0000","last_clean":"2024-03-11T18:31:00.458046+0000","last_became_active":"2024-03-11T18:30:27.268951+0000","last_became_peered":"2024-03-11T18:30:27.268951+0000","last_unstale":"2024-03-11T18:31:00.458046+0000","last_undegraded":"2024-03-11T18:31:00.458046+0000","last_fullsized":"2024-03-11T18:31:00.458046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:03:16.138585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057439+0000","last_change":"2024-03-11T18:30:32.309247+0000","last_active":"2024-03-11T18:30:34.057439+0000","last_peered":"2024-03-11T18:30:34.057439+0000","last_clean":"2024-03-11T18:30:34.057439+0000","last_became_active":"2024-03-11T18:30:32.308761+0000","last_became_peered":"2024-03-11T18:30:32.308761+0000","last_unstale":"2024-03-11T18:30:34.057439+0000","last_undegraded":"2024-03-11T18:30:34.057439+0000","last_fullsized":"2024-03-11T18:30:34.057439+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:10.525154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.436989+0000","last_change":"2024-03-11T18:30:27.279543+0000","last_active":"2024-03-11T18:30:59.436989+0000","last_peered":"2024-03-11T18:30:59.436989+0000","last_clean":"2024-03-11T18:30:59.436989+0000","last_became_active":"2024-03-11T18:30:27.279203+0000","last_became_peered":"2024-03-11T18:30:27.279203+0000","last_unstale":"2024-03-11T18:30:59.436989+0000","last_undegraded":"2024-03-11T18:30:59.436989+0000","last_fullsized":"2024-03-11T18:30:59.436989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:31.792757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.064060+0000","last_change":"2024-03-11T18:30:28.060508+0000","last_active":"2024-03-11T18:30:34.064060+0000","last_peered":"2024-03-11T18:30:34.064060+0000","last_clean":"2024-03-11T18:30:34.064060+0000","last_became_active":"2024-03-11T18:30:28.059074+0000","last_became_peered":"2024-03-11T18:30:28.059074+0000","last_unstale":"2024-03-11T18:30:34.064060+0000","last_undegraded":"2024-03-11T18:30:34.064060+0000","last_fullsized":"2024-03-11T18:30:34.064060+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:34.204704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059426+0000","last_change":"2024-03-11T18:30:32.528017+0000","last_active":"2024-03-11T18:30:34.059426+0000","last_peered":"2024-03-11T18:30:34.059426+0000","last_clean":"2024-03-11T18:30:34.059426+0000","last_became_active":"2024-03-11T18:30:32.527518+0000","last_became_peered":"2024-03-11T18:30:32.527518+0000","last_unstale":"2024-03-11T18:30:34.059426+0000","last_undegraded":"2024-03-11T18:30:34.059426+0000","last_fullsized":"2024-03-11T18:30:34.059426+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:13:09.095889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465943+0000","last_change":"2024-03-11T18:30:27.275540+0000","last_active":"2024-03-11T18:31:00.465943+0000","last_peered":"2024-03-11T18:31:00.465943+0000","last_clean":"2024-03-11T18:31:00.465943+0000","last_became_active":"2024-03-11T18:30:27.275070+0000","last_became_peered":"2024-03-11T18:30:27.275070+0000","last_unstale":"2024-03-11T18:31:00.465943+0000","last_undegraded":"2024-03-11T18:31:00.465943+0000","last_fullsized":"2024-03-11T18:31:00.465943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:59.729542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063246+0000","last_change":"2024-03-11T18:30:28.045096+0000","last_active":"2024-03-11T18:30:34.063246+0000","last_peered":"2024-03-11T18:30:34.063246+0000","last_clean":"2024-03-11T18:30:34.063246+0000","last_became_active":"2024-03-11T18:30:28.044408+0000","last_became_peered":"2024-03-11T18:30:28.044408+0000","last_unstale":"2024-03-11T18:30:34.063246+0000","last_undegraded":"2024-03-11T18:30:34.063246+0000","last_fullsized":"2024-03-11T18:30:34.063246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:49:59.426247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842052+0000","last_change":"2024-03-11T18:30:32.303288+0000","last_active":"2024-03-11T18:30:32.842052+0000","last_peered":"2024-03-11T18:30:32.842052+0000","last_clean":"2024-03-11T18:30:32.842052+0000","last_became_active":"2024-03-11T18:30:32.302746+0000","last_became_peered":"2024-03-11T18:30:32.302746+0000","last_unstale":"2024-03-11T18:30:32.842052+0000","last_undegraded":"2024-03-11T18:30:32.842052+0000","last_fullsized":"2024-03-11T18:30:32.842052+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:24.541448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.434794+0000","last_change":"2024-03-11T18:30:27.272880+0000","last_active":"2024-03-11T18:30:59.434794+0000","last_peered":"2024-03-11T18:30:59.434794+0000","last_clean":"2024-03-11T18:30:59.434794+0000","last_became_active":"2024-03-11T18:30:27.272611+0000","last_became_peered":"2024-03-11T18:30:27.272611+0000","last_unstale":"2024-03-11T18:30:59.434794+0000","last_undegraded":"2024-03-11T18:30:59.434794+0000","last_fullsized":"2024-03-11T18:30:59.434794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:41:10.322990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844385+0000","last_change":"2024-03-11T18:30:28.065613+0000","last_active":"2024-03-11T18:30:32.844385+0000","last_peered":"2024-03-11T18:30:32.844385+0000","last_clean":"2024-03-11T18:30:32.844385+0000","last_became_active":"2024-03-11T18:30:28.060520+0000","last_became_peered":"2024-03-11T18:30:28.060520+0000","last_unstale":"2024-03-11T18:30:32.844385+0000","last_undegraded":"2024-03-11T18:30:32.844385+0000","last_fullsized":"2024-03-11T18:30:32.844385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:44.530168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845179+0000","last_change":"2024-03-11T18:30:32.528633+0000","last_active":"2024-03-11T18:30:32.845179+0000","last_peered":"2024-03-11T18:30:32.845179+0000","last_clean":"2024-03-11T18:30:32.845179+0000","last_became_active":"2024-03-11T18:30:32.528073+0000","last_became_peered":"2024-03-11T18:30:32.528073+0000","last_unstale":"2024-03-11T18:30:32.845179+0000","last_undegraded":"2024-03-11T18:30:32.845179+0000","last_fullsized":"2024-03-11T18:30:32.845179+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:38.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432059+0000","last_change":"2024-03-11T18:30:27.261435+0000","last_active":"2024-03-11T18:30:59.432059+0000","last_peered":"2024-03-11T18:30:59.432059+0000","last_clean":"2024-03-11T18:30:59.432059+0000","last_became_active":"2024-03-11T18:30:27.260070+0000","last_became_peered":"2024-03-11T18:30:27.260070+0000","last_unstale":"2024-03-11T18:30:59.432059+0000","last_undegraded":"2024-03-11T18:30:59.432059+0000","last_fullsized":"2024-03-11T18:30:59.432059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:11.785818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055234+0000","last_change":"2024-03-11T18:30:28.041708+0000","last_active":"2024-03-11T18:30:34.055234+0000","last_peered":"2024-03-11T18:30:34.055234+0000","last_clean":"2024-03-11T18:30:34.055234+0000","last_became_active":"2024-03-11T18:30:28.039647+0000","last_became_peered":"2024-03-11T18:30:28.039647+0000","last_unstale":"2024-03-11T18:30:34.055234+0000","last_undegraded":"2024-03-11T18:30:34.055234+0000","last_fullsized":"2024-03-11T18:30:34.055234+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.624175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058652+0000","last_change":"2024-03-11T18:30:32.326512+0000","last_active":"2024-03-11T18:30:34.058652+0000","last_peered":"2024-03-11T18:30:34.058652+0000","last_clean":"2024-03-11T18:30:34.058652+0000","last_became_active":"2024-03-11T18:30:32.325253+0000","last_became_peered":"2024-03-11T18:30:32.325253+0000","last_unstale":"2024-03-11T18:30:34.058652+0000","last_undegraded":"2024-03-11T18:30:34.058652+0000","last_fullsized":"2024-03-11T18:30:34.058652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:48.589769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.435979+0000","last_change":"2024-03-11T18:30:27.286194+0000","last_active":"2024-03-11T18:30:59.435979+0000","last_peered":"2024-03-11T18:30:59.435979+0000","last_clean":"2024-03-11T18:30:59.435979+0000","last_became_active":"2024-03-11T18:30:27.284877+0000","last_became_peered":"2024-03-11T18:30:27.284877+0000","last_unstale":"2024-03-11T18:30:59.435979+0000","last_undegraded":"2024-03-11T18:30:59.435979+0000","last_fullsized":"2024-03-11T18:30:59.435979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:56:20.324266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056625+0000","last_change":"2024-03-11T18:30:28.061038+0000","last_active":"2024-03-11T18:30:34.056625+0000","last_peered":"2024-03-11T18:30:34.056625+0000","last_clean":"2024-03-11T18:30:34.056625+0000","last_became_active":"2024-03-11T18:30:28.060701+0000","last_became_peered":"2024-03-11T18:30:28.060701+0000","last_unstale":"2024-03-11T18:30:34.056625+0000","last_undegraded":"2024-03-11T18:30:34.056625+0000","last_fullsized":"2024-03-11T18:30:34.056625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:11.838309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845215+0000","last_change":"2024-03-11T18:30:32.316762+0000","last_active":"2024-03-11T18:30:32.845215+0000","last_peered":"2024-03-11T18:30:32.845215+0000","last_clean":"2024-03-11T18:30:32.845215+0000","last_became_active":"2024-03-11T18:30:32.316318+0000","last_became_peered":"2024-03-11T18:30:32.316318+0000","last_unstale":"2024-03-11T18:30:32.845215+0000","last_undegraded":"2024-03-11T18:30:32.845215+0000","last_fullsized":"2024-03-11T18:30:32.845215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:29.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.841658+0000","last_change":"2024-03-11T18:30:27.276636+0000","last_active":"2024-03-11T18:30:32.841658+0000","last_peered":"2024-03-11T18:30:32.841658+0000","last_clean":"2024-03-11T18:30:32.841658+0000","last_became_active":"2024-03-11T18:30:27.276290+0000","last_became_peered":"2024-03-11T18:30:27.276290+0000","last_unstale":"2024-03-11T18:30:32.841658+0000","last_undegraded":"2024-03-11T18:30:32.841658+0000","last_fullsized":"2024-03-11T18:30:32.841658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:34.272568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.062534+0000","last_change":"2024-03-11T18:30:28.067822+0000","last_active":"2024-03-11T18:30:34.062534+0000","last_peered":"2024-03-11T18:30:34.062534+0000","last_clean":"2024-03-11T18:30:34.062534+0000","last_became_active":"2024-03-11T18:30:28.067379+0000","last_became_peered":"2024-03-11T18:30:28.067379+0000","last_unstale":"2024-03-11T18:30:34.062534+0000","last_undegraded":"2024-03-11T18:30:34.062534+0000","last_fullsized":"2024-03-11T18:30:34.062534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:27.730186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844905+0000","last_change":"2024-03-11T18:30:32.527845+0000","last_active":"2024-03-11T18:30:32.844905+0000","last_peered":"2024-03-11T18:30:32.844905+0000","last_clean":"2024-03-11T18:30:32.844905+0000","last_became_active":"2024-03-11T18:30:32.527356+0000","last_became_peered":"2024-03-11T18:30:32.527356+0000","last_unstale":"2024-03-11T18:30:32.844905+0000","last_undegraded":"2024-03-11T18:30:32.844905+0000","last_fullsized":"2024-03-11T18:30:32.844905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.005118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432713+0000","last_change":"2024-03-11T18:30:27.267263+0000","last_active":"2024-03-11T18:30:59.432713+0000","last_peered":"2024-03-11T18:30:59.432713+0000","last_clean":"2024-03-11T18:30:59.432713+0000","last_became_active":"2024-03-11T18:30:27.266893+0000","last_became_peered":"2024-03-11T18:30:27.266893+0000","last_unstale":"2024-03-11T18:30:59.432713+0000","last_undegraded":"2024-03-11T18:30:59.432713+0000","last_fullsized":"2024-03-11T18:30:59.432713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:04.305623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962566+0000","last_change":"2024-03-11T18:30:28.054211+0000","last_active":"2024-03-11T18:30:33.962566+0000","last_peered":"2024-03-11T18:30:33.962566+0000","last_clean":"2024-03-11T18:30:33.962566+0000","last_became_active":"2024-03-11T18:30:28.053472+0000","last_became_peered":"2024-03-11T18:30:28.053472+0000","last_unstale":"2024-03-11T18:30:33.962566+0000","last_undegraded":"2024-03-11T18:30:33.962566+0000","last_fullsized":"2024-03-11T18:30:33.962566+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:37.415224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.054931+0000","last_change":"2024-03-11T18:30:32.304049+0000","last_active":"2024-03-11T18:30:34.054931+0000","last_peered":"2024-03-11T18:30:34.054931+0000","last_clean":"2024-03-11T18:30:34.054931+0000","last_became_active":"2024-03-11T18:30:32.303410+0000","last_became_peered":"2024-03-11T18:30:32.303410+0000","last_unstale":"2024-03-11T18:30:34.054931+0000","last_undegraded":"2024-03-11T18:30:34.054931+0000","last_fullsized":"2024-03-11T18:30:34.054931+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:15.574274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.458118+0000","last_change":"2024-03-11T18:30:27.263269+0000","last_active":"2024-03-11T18:31:00.458118+0000","last_peered":"2024-03-11T18:31:00.458118+0000","last_clean":"2024-03-11T18:31:00.458118+0000","last_became_active":"2024-03-11T18:30:27.262892+0000","last_became_peered":"2024-03-11T18:30:27.262892+0000","last_unstale":"2024-03-11T18:31:00.458118+0000","last_undegraded":"2024-03-11T18:31:00.458118+0000","last_fullsized":"2024-03-11T18:31:00.458118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:59.753276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963108+0000","last_change":"2024-03-11T18:30:28.043362+0000","last_active":"2024-03-11T18:30:33.963108+0000","last_peered":"2024-03-11T18:30:33.963108+0000","last_clean":"2024-03-11T18:30:33.963108+0000","last_became_active":"2024-03-11T18:30:28.042566+0000","last_became_peered":"2024-03-11T18:30:28.042566+0000","last_unstale":"2024-03-11T18:30:33.963108+0000","last_undegraded":"2024-03-11T18:30:33.963108+0000","last_fullsized":"2024-03-11T18:30:33.963108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:41.226647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843110+0000","last_change":"2024-03-11T18:30:32.323886+0000","last_active":"2024-03-11T18:30:32.843110+0000","last_peered":"2024-03-11T18:30:32.843110+0000","last_clean":"2024-03-11T18:30:32.843110+0000","last_became_active":"2024-03-11T18:30:32.321785+0000","last_became_peered":"2024-03-11T18:30:32.321785+0000","last_unstale":"2024-03-11T18:30:32.843110+0000","last_undegraded":"2024-03-11T18:30:32.843110+0000","last_fullsized":"2024-03-11T18:30:32.843110+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:25.411440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056308+0000","last_change":"2024-03-11T18:30:27.271415+0000","last_active":"2024-03-11T18:30:34.056308+0000","last_peered":"2024-03-11T18:30:34.056308+0000","last_clean":"2024-03-11T18:30:34.056308+0000","last_became_active":"2024-03-11T18:30:27.271041+0000","last_became_peered":"2024-03-11T18:30:27.271041+0000","last_unstale":"2024-03-11T18:30:34.056308+0000","last_undegraded":"2024-03-11T18:30:34.056308+0000","last_fullsized":"2024-03-11T18:30:34.056308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:06.967984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962881+0000","last_change":"2024-03-11T18:30:28.042918+0000","last_active":"2024-03-11T18:30:33.962881+0000","last_peered":"2024-03-11T18:30:33.962881+0000","last_clean":"2024-03-11T18:30:33.962881+0000","last_became_active":"2024-03-11T18:30:28.042493+0000","last_became_peered":"2024-03-11T18:30:28.042493+0000","last_unstale":"2024-03-11T18:30:33.962881+0000","last_undegraded":"2024-03-11T18:30:33.962881+0000","last_fullsized":"2024-03-11T18:30:33.962881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:32:57.501997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055541+0000","last_change":"2024-03-11T18:30:32.304111+0000","last_active":"2024-03-11T18:30:34.055541+0000","last_peered":"2024-03-11T18:30:34.055541+0000","last_clean":"2024-03-11T18:30:34.055541+0000","last_became_active":"2024-03-11T18:30:32.303420+0000","last_became_peered":"2024-03-11T18:30:32.303420+0000","last_unstale":"2024-03-11T18:30:34.055541+0000","last_undegraded":"2024-03-11T18:30:34.055541+0000","last_fullsized":"2024-03-11T18:30:34.055541+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:51.801772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061534+0000","last_change":"2024-03-11T18:30:27.288015+0000","last_active":"2024-03-11T18:30:34.061534+0000","last_peered":"2024-03-11T18:30:34.061534+0000","last_clean":"2024-03-11T18:30:34.061534+0000","last_became_active":"2024-03-11T18:30:27.287528+0000","last_became_peered":"2024-03-11T18:30:27.287528+0000","last_unstale":"2024-03-11T18:30:34.061534+0000","last_undegraded":"2024-03-11T18:30:34.061534+0000","last_fullsized":"2024-03-11T18:30:34.061534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:56:33.612120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962910+0000","last_change":"2024-03-11T18:30:28.041888+0000","last_active":"2024-03-11T18:30:33.962910+0000","last_peered":"2024-03-11T18:30:33.962910+0000","last_clean":"2024-03-11T18:30:33.962910+0000","last_became_active":"2024-03-11T18:30:28.041506+0000","last_became_peered":"2024-03-11T18:30:28.041506+0000","last_unstale":"2024-03-11T18:30:33.962910+0000","last_undegraded":"2024-03-11T18:30:33.962910+0000","last_fullsized":"2024-03-11T18:30:33.962910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:46.308525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056752+0000","last_change":"2024-03-11T18:30:28.037778+0000","last_active":"2024-03-11T18:30:34.056752+0000","last_peered":"2024-03-11T18:30:34.056752+0000","last_clean":"2024-03-11T18:30:34.056752+0000","last_became_active":"2024-03-11T18:30:28.037428+0000","last_became_peered":"2024-03-11T18:30:28.037428+0000","last_unstale":"2024-03-11T18:30:34.056752+0000","last_undegraded":"2024-03-11T18:30:34.056752+0000","last_fullsized":"2024-03-11T18:30:34.056752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:30:35.810425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842265+0000","last_change":"2024-03-11T18:30:28.037380+0000","last_active":"2024-03-11T18:30:32.842265+0000","last_peered":"2024-03-11T18:30:32.842265+0000","last_clean":"2024-03-11T18:30:32.842265+0000","last_became_active":"2024-03-11T18:30:28.037028+0000","last_became_peered":"2024-03-11T18:30:28.037028+0000","last_unstale":"2024-03-11T18:30:32.842265+0000","last_undegraded":"2024-03-11T18:30:32.842265+0000","last_fullsized":"2024-03-11T18:30:32.842265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:34.161889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.841615+0000","last_change":"2024-03-11T18:30:28.060947+0000","last_active":"2024-03-11T18:30:32.841615+0000","last_peered":"2024-03-11T18:30:32.841615+0000","last_clean":"2024-03-11T18:30:32.841615+0000","last_became_active":"2024-03-11T18:30:28.060600+0000","last_became_peered":"2024-03-11T18:30:28.060600+0000","last_unstale":"2024-03-11T18:30:32.841615+0000","last_undegraded":"2024-03-11T18:30:32.841615+0000","last_fullsized":"2024-03-11T18:30:32.841615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:31.964347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843351+0000","last_change":"2024-03-11T18:30:28.062110+0000","last_active":"2024-03-11T18:30:32.843351+0000","last_peered":"2024-03-11T18:30:32.843351+0000","last_clean":"2024-03-11T18:30:32.843351+0000","last_became_active":"2024-03-11T18:30:28.061427+0000","last_became_peered":"2024-03-11T18:30:28.061427+0000","last_unstale":"2024-03-11T18:30:32.843351+0000","last_undegraded":"2024-03-11T18:30:32.843351+0000","last_fullsized":"2024-03-11T18:30:32.843351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:32.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063886+0000","last_change":"2024-03-11T18:30:28.060118+0000","last_active":"2024-03-11T18:30:34.063886+0000","last_peered":"2024-03-11T18:30:34.063886+0000","last_clean":"2024-03-11T18:30:34.063886+0000","last_became_active":"2024-03-11T18:30:28.059733+0000","last_became_peered":"2024-03-11T18:30:28.059733+0000","last_unstale":"2024-03-11T18:30:34.063886+0000","last_undegraded":"2024-03-11T18:30:34.063886+0000","last_fullsized":"2024-03-11T18:30:34.063886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:18:56.059082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057307+0000","last_change":"2024-03-11T18:30:28.041817+0000","last_active":"2024-03-11T18:30:34.057307+0000","last_peered":"2024-03-11T18:30:34.057307+0000","last_clean":"2024-03-11T18:30:34.057307+0000","last_became_active":"2024-03-11T18:30:28.041346+0000","last_became_peered":"2024-03-11T18:30:28.041346+0000","last_unstale":"2024-03-11T18:30:34.057307+0000","last_undegraded":"2024-03-11T18:30:34.057307+0000","last_fullsized":"2024-03-11T18:30:34.057307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:26:28.544602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842271+0000","last_change":"2024-03-11T18:30:28.040820+0000","last_active":"2024-03-11T18:30:32.842271+0000","last_peered":"2024-03-11T18:30:32.842271+0000","last_clean":"2024-03-11T18:30:32.842271+0000","last_became_active":"2024-03-11T18:30:28.040381+0000","last_became_peered":"2024-03-11T18:30:28.040381+0000","last_unstale":"2024-03-11T18:30:32.842271+0000","last_undegraded":"2024-03-11T18:30:32.842271+0000","last_fullsized":"2024-03-11T18:30:32.842271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:33:37.120786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063449+0000","last_change":"2024-03-11T18:30:28.052050+0000","last_active":"2024-03-11T18:30:34.063449+0000","last_peered":"2024-03-11T18:30:34.063449+0000","last_clean":"2024-03-11T18:30:34.063449+0000","last_became_active":"2024-03-11T18:30:28.051685+0000","last_became_peered":"2024-03-11T18:30:28.051685+0000","last_unstale":"2024-03-11T18:30:34.063449+0000","last_undegraded":"2024-03-11T18:30:34.063449+0000","last_fullsized":"2024-03-11T18:30:34.063449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:02.777603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055831+0000","last_change":"2024-03-11T18:30:28.053993+0000","last_active":"2024-03-11T18:30:34.055831+0000","last_peered":"2024-03-11T18:30:34.055831+0000","last_clean":"2024-03-11T18:30:34.055831+0000","last_became_active":"2024-03-11T18:30:28.053678+0000","last_became_peered":"2024-03-11T18:30:28.053678+0000","last_unstale":"2024-03-11T18:30:34.055831+0000","last_undegraded":"2024-03-11T18:30:34.055831+0000","last_fullsized":"2024-03-11T18:30:34.055831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:48.801078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059246+0000","last_change":"2024-03-11T18:30:28.051357+0000","last_active":"2024-03-11T18:30:34.059246+0000","last_peered":"2024-03-11T18:30:34.059246+0000","last_clean":"2024-03-11T18:30:34.059246+0000","last_became_active":"2024-03-11T18:30:28.051046+0000","last_became_peered":"2024-03-11T18:30:28.051046+0000","last_unstale":"2024-03-11T18:30:34.059246+0000","last_undegraded":"2024-03-11T18:30:34.059246+0000","last_fullsized":"2024-03-11T18:30:34.059246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:11.994754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055143+0000","last_change":"2024-03-11T18:30:28.051628+0000","last_active":"2024-03-11T18:30:34.055143+0000","last_peered":"2024-03-11T18:30:34.055143+0000","last_clean":"2024-03-11T18:30:34.055143+0000","last_became_active":"2024-03-11T18:30:28.051129+0000","last_became_peered":"2024-03-11T18:30:28.051129+0000","last_unstale":"2024-03-11T18:30:34.055143+0000","last_undegraded":"2024-03-11T18:30:34.055143+0000","last_fullsized":"2024-03-11T18:30:34.055143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:47.568747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060554+0000","last_change":"2024-03-11T18:30:28.065792+0000","last_active":"2024-03-11T18:30:34.060554+0000","last_peered":"2024-03-11T18:30:34.060554+0000","last_clean":"2024-03-11T18:30:34.060554+0000","last_became_active":"2024-03-11T18:30:28.064902+0000","last_became_peered":"2024-03-11T18:30:28.064902+0000","last_unstale":"2024-03-11T18:30:34.060554+0000","last_undegraded":"2024-03-11T18:30:34.060554+0000","last_fullsized":"2024-03-11T18:30:34.060554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:18.019872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057149+0000","last_change":"2024-03-11T18:30:28.041663+0000","last_active":"2024-03-11T18:30:34.057149+0000","last_peered":"2024-03-11T18:30:34.057149+0000","last_clean":"2024-03-11T18:30:34.057149+0000","last_became_active":"2024-03-11T18:30:28.041257+0000","last_became_peered":"2024-03-11T18:30:28.041257+0000","last_unstale":"2024-03-11T18:30:34.057149+0000","last_undegraded":"2024-03-11T18:30:34.057149+0000","last_fullsized":"2024-03-11T18:30:34.057149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:18.216295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7843,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":15,"num_read_kb":13,"num_write":126,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":28835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318291,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1079062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":484714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":484634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266731,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1609728,"data_stored":1076838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430256,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":487738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":484590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":489088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":484090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855420,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":484540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691894,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1075740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":10442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T18:31:05.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-11T18:31:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.712+0000 7f17ee800640 1 -- 172.21.15.2:0/260330007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 msgr2=0x7f17e8106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.712+0000 7f17ee800640 1 --2- 172.21.15.2:0/260330007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8106830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009900 tx=0x7f17d402bc30 comp rx=0 tx=0).stop 2024-03-11T18:31:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.712+0000 7f17ee800640 1 -- 172.21.15.2:0/260330007 shutdown_connections 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 --2- 172.21.15.2:0/260330007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8106d70 0x7f17e81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 --2- 172.21.15.2:0/260330007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8106830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 --2- 172.21.15.2:0/260330007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8101b10 0x7f17e8103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 -- 172.21.15.2:0/260330007 >> 172.21.15.2:0/260330007 conn(0x7f17e80fb500 msgr2=0x7f17e80fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 -- 172.21.15.2:0/260330007 shutdown_connections 2024-03-11T18:31:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.713+0000 7f17ee800640 1 -- 172.21.15.2:0/260330007 wait complete. 2024-03-11T18:31:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 Processor -- start 2024-03-11T18:31:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 -- start start 2024-03-11T18:31:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 0x7f17e81971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8106d70 0x7f17e819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819ec90 con 0x7f17e8101b10 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.714+0000 7f17ee800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819ee00 con 0x7f17e8104440 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17ee800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e819efc0 con 0x7f17e8106d70 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17ed400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 0x7f17e81971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47462/0 (socket says 172.21.15.2:47462) 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 -- 172.21.15.2:0/2802594175 learned_addr learned my addr 172.21.15.2:0/2802594175 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17ede00640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8106d70 0x7f17e819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 -- 172.21.15.2:0/2802594175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8106d70 msgr2=0x7f17e819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8106d70 0x7f17e819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 -- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 msgr2=0x7f17e81971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 0x7f17e81971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17eca00640 1 -- 172.21.15.2:0/2802594175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d8009490 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.715+0000 7f17ed400640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 0x7f17e81971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17eca00640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17d4005ab0 tx=0x7f17d4005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d40043d0 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17d40052d0 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4009560 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4003cd0 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.716+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e819f610 con 0x7f17e8104440 2024-03-11T18:31:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.717+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e80fcff0 con 0x7f17e8104440 2024-03-11T18:31:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.719+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f17d4005440 con 0x7f17e8104440 2024-03-11T18:31:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.721+0000 7f17e6000640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 0x7f17cc0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.721+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f17d40ef740 con 0x7f17e8104440 2024-03-11T18:31:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.721+0000 7f17ed400640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 0x7f17cc0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:05.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.722+0000 7f17ed400640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 0x7f17cc0a3ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005fd0 tx=0x7f17e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:05.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.725+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f17d40b86c0 con 0x7f17e8104440 2024-03-11T18:31:05.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:05 smithi002 ceph-mon[31328]: pgmap v339: 129 pgs: 129 active+clean; 584 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:05.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:05 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1069052964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:05 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/328589094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:05 smithi023 ceph-mon[36406]: pgmap v339: 129 pgs: 129 active+clean; 584 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:05.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:05 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1069052964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:05 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/328589094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:05 smithi165 ceph-mon[36634]: pgmap v339: 129 pgs: 129 active+clean; 584 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:05 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1069052964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:05 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/328589094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:05.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.923+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17e802ce10 con 0x7f17cc0a1b30 2024-03-11T18:31:05.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.948+0000 7f17e6000640 1 -- 172.21.15.2:0/2802594175 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340358 (secure 0 0 0) 0x7f17e802ce10 con 0x7f17cc0a1b30 2024-03-11T18:31:05.954 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:05.956 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.954+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 msgr2=0x7f17cc0a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.954+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 0x7f17cc0a3ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005fd0 tx=0x7f17e0009500 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.954+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 msgr2=0x7f17e8197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.954+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f17d4005ab0 tx=0x7f17d4005e50 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 shutdown_connections 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17cc0a1b30 0x7f17cc0a3ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e8106d70 0x7f17e819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17e8104440 0x7f17e8197730 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 --2- 172.21.15.2:0/2802594175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17e8101b10 0x7f17e81971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 >> 172.21.15.2:0/2802594175 conn(0x7f17e80fb500 msgr2=0x7f17e8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 shutdown_connections 2024-03-11T18:31:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:05.955+0000 7f17ee800640 1 -- 172.21.15.2:0/2802594175 wait complete. 2024-03-11T18:31:05.975 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-11T18:31:04.211195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598328,"num_objects":61,"num_object_clones":0,"num_object_copies":184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":151,"num_read_kb":132,"num_write":308,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366976,"kb_used_data":13660,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574848,"statfs":{"total":1151940427776,"available":1151564644352,"internally_reserved":0,"allocated":13987840,"data_stored":7598916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3461,"num_objects":37,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":15,"num_read_kb":13,"num_write":79,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006628"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057433+0000","last_change":"2024-03-11T18:30:28.060909+0000","last_active":"2024-03-11T18:30:34.057433+0000","last_peered":"2024-03-11T18:30:34.057433+0000","last_clean":"2024-03-11T18:30:34.057433+0000","last_became_active":"2024-03-11T18:30:28.060554+0000","last_became_peered":"2024-03-11T18:30:28.060554+0000","last_unstale":"2024-03-11T18:30:34.057433+0000","last_undegraded":"2024-03-11T18:30:34.057433+0000","last_fullsized":"2024-03-11T18:30:34.057433+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:33.609820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844312+0000","last_change":"2024-03-11T18:30:28.055136+0000","last_active":"2024-03-11T18:30:32.844312+0000","last_peered":"2024-03-11T18:30:32.844312+0000","last_clean":"2024-03-11T18:30:32.844312+0000","last_became_active":"2024-03-11T18:30:28.054785+0000","last_became_peered":"2024-03-11T18:30:28.054785+0000","last_unstale":"2024-03-11T18:30:32.844312+0000","last_undegraded":"2024-03-11T18:30:32.844312+0000","last_fullsized":"2024-03-11T18:30:32.844312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:12.702850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843458+0000","last_change":"2024-03-11T18:30:28.051593+0000","last_active":"2024-03-11T18:30:32.843458+0000","last_peered":"2024-03-11T18:30:32.843458+0000","last_clean":"2024-03-11T18:30:32.843458+0000","last_became_active":"2024-03-11T18:30:28.049344+0000","last_became_peered":"2024-03-11T18:30:28.049344+0000","last_unstale":"2024-03-11T18:30:32.843458+0000","last_undegraded":"2024-03-11T18:30:32.843458+0000","last_fullsized":"2024-03-11T18:30:32.843458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:06.433202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845213+0000","last_change":"2024-03-11T18:30:28.036335+0000","last_active":"2024-03-11T18:30:32.845213+0000","last_peered":"2024-03-11T18:30:32.845213+0000","last_clean":"2024-03-11T18:30:32.845213+0000","last_became_active":"2024-03-11T18:30:28.035728+0000","last_became_peered":"2024-03-11T18:30:28.035728+0000","last_unstale":"2024-03-11T18:30:32.845213+0000","last_undegraded":"2024-03-11T18:30:32.845213+0000","last_fullsized":"2024-03-11T18:30:32.845213+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:10.728266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055316+0000","last_change":"2024-03-11T18:30:28.050560+0000","last_active":"2024-03-11T18:30:34.055316+0000","last_peered":"2024-03-11T18:30:34.055316+0000","last_clean":"2024-03-11T18:30:34.055316+0000","last_became_active":"2024-03-11T18:30:28.050273+0000","last_became_peered":"2024-03-11T18:30:28.050273+0000","last_unstale":"2024-03-11T18:30:34.055316+0000","last_undegraded":"2024-03-11T18:30:34.055316+0000","last_fullsized":"2024-03-11T18:30:34.055316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:56.780575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432062+0000","last_change":"2024-03-11T18:30:28.056879+0000","last_active":"2024-03-11T18:30:33.432062+0000","last_peered":"2024-03-11T18:30:33.432062+0000","last_clean":"2024-03-11T18:30:33.432062+0000","last_became_active":"2024-03-11T18:30:28.056435+0000","last_became_peered":"2024-03-11T18:30:28.056435+0000","last_unstale":"2024-03-11T18:30:33.432062+0000","last_undegraded":"2024-03-11T18:30:33.432062+0000","last_fullsized":"2024-03-11T18:30:33.432062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:04.595503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962340+0000","last_change":"2024-03-11T18:30:28.044355+0000","last_active":"2024-03-11T18:30:33.962340+0000","last_peered":"2024-03-11T18:30:33.962340+0000","last_clean":"2024-03-11T18:30:33.962340+0000","last_became_active":"2024-03-11T18:30:28.043875+0000","last_became_peered":"2024-03-11T18:30:28.043875+0000","last_unstale":"2024-03-11T18:30:33.962340+0000","last_undegraded":"2024-03-11T18:30:33.962340+0000","last_fullsized":"2024-03-11T18:30:33.962340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:25.924359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059316+0000","last_change":"2024-03-11T18:30:28.055633+0000","last_active":"2024-03-11T18:30:34.059316+0000","last_peered":"2024-03-11T18:30:34.059316+0000","last_clean":"2024-03-11T18:30:34.059316+0000","last_became_active":"2024-03-11T18:30:28.055158+0000","last_became_peered":"2024-03-11T18:30:28.055158+0000","last_unstale":"2024-03-11T18:30:34.059316+0000","last_undegraded":"2024-03-11T18:30:34.059316+0000","last_fullsized":"2024-03-11T18:30:34.059316+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:21:45.379551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055468+0000","last_change":"2024-03-11T18:30:28.037306+0000","last_active":"2024-03-11T18:30:34.055468+0000","last_peered":"2024-03-11T18:30:34.055468+0000","last_clean":"2024-03-11T18:30:34.055468+0000","last_became_active":"2024-03-11T18:30:28.036930+0000","last_became_peered":"2024-03-11T18:30:28.036930+0000","last_unstale":"2024-03-11T18:30:34.055468+0000","last_undegraded":"2024-03-11T18:30:34.055468+0000","last_fullsized":"2024-03-11T18:30:34.055468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432512+0000","last_change":"2024-03-11T18:30:28.038914+0000","last_active":"2024-03-11T18:30:33.432512+0000","last_peered":"2024-03-11T18:30:33.432512+0000","last_clean":"2024-03-11T18:30:33.432512+0000","last_became_active":"2024-03-11T18:30:28.038266+0000","last_became_peered":"2024-03-11T18:30:28.038266+0000","last_unstale":"2024-03-11T18:30:33.432512+0000","last_undegraded":"2024-03-11T18:30:33.432512+0000","last_fullsized":"2024-03-11T18:30:33.432512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:59.136860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963101+0000","last_change":"2024-03-11T18:30:28.041833+0000","last_active":"2024-03-11T18:30:33.963101+0000","last_peered":"2024-03-11T18:30:33.963101+0000","last_clean":"2024-03-11T18:30:33.963101+0000","last_became_active":"2024-03-11T18:30:28.041307+0000","last_became_peered":"2024-03-11T18:30:28.041307+0000","last_unstale":"2024-03-11T18:30:33.963101+0000","last_undegraded":"2024-03-11T18:30:33.963101+0000","last_fullsized":"2024-03-11T18:30:33.963101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:35.803202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845101+0000","last_change":"2024-03-11T18:30:28.037646+0000","last_active":"2024-03-11T18:30:32.845101+0000","last_peered":"2024-03-11T18:30:32.845101+0000","last_clean":"2024-03-11T18:30:32.845101+0000","last_became_active":"2024-03-11T18:30:28.037272+0000","last_became_peered":"2024-03-11T18:30:28.037272+0000","last_unstale":"2024-03-11T18:30:32.845101+0000","last_undegraded":"2024-03-11T18:30:32.845101+0000","last_fullsized":"2024-03-11T18:30:32.845101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:50.594666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060405+0000","last_change":"2024-03-11T18:30:28.056917+0000","last_active":"2024-03-11T18:30:34.060405+0000","last_peered":"2024-03-11T18:30:34.060405+0000","last_clean":"2024-03-11T18:30:34.060405+0000","last_became_active":"2024-03-11T18:30:28.056496+0000","last_became_peered":"2024-03-11T18:30:28.056496+0000","last_unstale":"2024-03-11T18:30:34.060405+0000","last_undegraded":"2024-03-11T18:30:34.060405+0000","last_fullsized":"2024-03-11T18:30:34.060405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:44.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842819+0000","last_change":"2024-03-11T18:30:28.055180+0000","last_active":"2024-03-11T18:30:32.842819+0000","last_peered":"2024-03-11T18:30:32.842819+0000","last_clean":"2024-03-11T18:30:32.842819+0000","last_became_active":"2024-03-11T18:30:28.054807+0000","last_became_peered":"2024-03-11T18:30:28.054807+0000","last_unstale":"2024-03-11T18:30:32.842819+0000","last_undegraded":"2024-03-11T18:30:32.842819+0000","last_fullsized":"2024-03-11T18:30:32.842819+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:31.671461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842796+0000","last_change":"2024-03-11T18:30:28.065015+0000","last_active":"2024-03-11T18:30:32.842796+0000","last_peered":"2024-03-11T18:30:32.842796+0000","last_clean":"2024-03-11T18:30:32.842796+0000","last_became_active":"2024-03-11T18:30:28.061131+0000","last_became_peered":"2024-03-11T18:30:28.061131+0000","last_unstale":"2024-03-11T18:30:32.842796+0000","last_undegraded":"2024-03-11T18:30:32.842796+0000","last_fullsized":"2024-03-11T18:30:32.842796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:35.314562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.054769+0000","last_change":"2024-03-11T18:30:28.054388+0000","last_active":"2024-03-11T18:30:34.054769+0000","last_peered":"2024-03-11T18:30:34.054769+0000","last_clean":"2024-03-11T18:30:34.054769+0000","last_became_active":"2024-03-11T18:30:28.054036+0000","last_became_peered":"2024-03-11T18:30:28.054036+0000","last_unstale":"2024-03-11T18:30:34.054769+0000","last_undegraded":"2024-03-11T18:30:34.054769+0000","last_fullsized":"2024-03-11T18:30:34.054769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:45:21.476944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057326+0000","last_change":"2024-03-11T18:30:28.061935+0000","last_active":"2024-03-11T18:30:34.057326+0000","last_peered":"2024-03-11T18:30:34.057326+0000","last_clean":"2024-03-11T18:30:34.057326+0000","last_became_active":"2024-03-11T18:30:28.061558+0000","last_became_peered":"2024-03-11T18:30:28.061558+0000","last_unstale":"2024-03-11T18:30:34.057326+0000","last_undegraded":"2024-03-11T18:30:34.057326+0000","last_fullsized":"2024-03-11T18:30:34.057326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:01.728956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058341+0000","last_change":"2024-03-11T18:30:28.058372+0000","last_active":"2024-03-11T18:30:34.058341+0000","last_peered":"2024-03-11T18:30:34.058341+0000","last_clean":"2024-03-11T18:30:34.058341+0000","last_became_active":"2024-03-11T18:30:28.058028+0000","last_became_peered":"2024-03-11T18:30:28.058028+0000","last_unstale":"2024-03-11T18:30:34.058341+0000","last_undegraded":"2024-03-11T18:30:34.058341+0000","last_fullsized":"2024-03-11T18:30:34.058341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:20:15.645368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842261+0000","last_change":"2024-03-11T18:30:28.054845+0000","last_active":"2024-03-11T18:30:32.842261+0000","last_peered":"2024-03-11T18:30:32.842261+0000","last_clean":"2024-03-11T18:30:32.842261+0000","last_became_active":"2024-03-11T18:30:28.054497+0000","last_became_peered":"2024-03-11T18:30:28.054497+0000","last_unstale":"2024-03-11T18:30:32.842261+0000","last_undegraded":"2024-03-11T18:30:32.842261+0000","last_fullsized":"2024-03-11T18:30:32.842261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:52.100045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842262+0000","last_change":"2024-03-11T18:30:32.322103+0000","last_active":"2024-03-11T18:30:32.842262+0000","last_peered":"2024-03-11T18:30:32.842262+0000","last_clean":"2024-03-11T18:30:32.842262+0000","last_became_active":"2024-03-11T18:30:32.321422+0000","last_became_peered":"2024-03-11T18:30:32.321422+0000","last_unstale":"2024-03-11T18:30:32.842262+0000","last_undegraded":"2024-03-11T18:30:32.842262+0000","last_fullsized":"2024-03-11T18:30:32.842262+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:56.946318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.464119+0000","last_change":"2024-03-11T18:30:27.283463+0000","last_active":"2024-03-11T18:31:00.464119+0000","last_peered":"2024-03-11T18:31:00.464119+0000","last_clean":"2024-03-11T18:31:00.464119+0000","last_became_active":"2024-03-11T18:30:27.279100+0000","last_became_peered":"2024-03-11T18:30:27.279100+0000","last_unstale":"2024-03-11T18:31:00.464119+0000","last_undegraded":"2024-03-11T18:31:00.464119+0000","last_fullsized":"2024-03-11T18:31:00.464119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:38.096395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063150+0000","last_change":"2024-03-11T18:30:28.051288+0000","last_active":"2024-03-11T18:30:34.063150+0000","last_peered":"2024-03-11T18:30:34.063150+0000","last_clean":"2024-03-11T18:30:34.063150+0000","last_became_active":"2024-03-11T18:30:28.050726+0000","last_became_peered":"2024-03-11T18:30:28.050726+0000","last_unstale":"2024-03-11T18:30:34.063150+0000","last_undegraded":"2024-03-11T18:30:34.063150+0000","last_fullsized":"2024-03-11T18:30:34.063150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:29.125099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844657+0000","last_change":"2024-03-11T18:30:32.311188+0000","last_active":"2024-03-11T18:30:32.844657+0000","last_peered":"2024-03-11T18:30:32.844657+0000","last_clean":"2024-03-11T18:30:32.844657+0000","last_became_active":"2024-03-11T18:30:32.310670+0000","last_became_peered":"2024-03-11T18:30:32.310670+0000","last_unstale":"2024-03-11T18:30:32.844657+0000","last_undegraded":"2024-03-11T18:30:32.844657+0000","last_fullsized":"2024-03-11T18:30:32.844657+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:51:17.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063752+0000","last_change":"2024-03-11T18:30:27.284990+0000","last_active":"2024-03-11T18:30:34.063752+0000","last_peered":"2024-03-11T18:30:34.063752+0000","last_clean":"2024-03-11T18:30:34.063752+0000","last_became_active":"2024-03-11T18:30:27.283629+0000","last_became_peered":"2024-03-11T18:30:27.283629+0000","last_unstale":"2024-03-11T18:30:34.063752+0000","last_undegraded":"2024-03-11T18:30:34.063752+0000","last_fullsized":"2024-03-11T18:30:34.063752+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:33.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844512+0000","last_change":"2024-03-11T18:30:28.066345+0000","last_active":"2024-03-11T18:30:32.844512+0000","last_peered":"2024-03-11T18:30:32.844512+0000","last_clean":"2024-03-11T18:30:32.844512+0000","last_became_active":"2024-03-11T18:30:28.061654+0000","last_became_peered":"2024-03-11T18:30:28.061654+0000","last_unstale":"2024-03-11T18:30:32.844512+0000","last_undegraded":"2024-03-11T18:30:32.844512+0000","last_fullsized":"2024-03-11T18:30:32.844512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:29.651510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844691+0000","last_change":"2024-03-11T18:30:32.318088+0000","last_active":"2024-03-11T18:30:32.844691+0000","last_peered":"2024-03-11T18:30:32.844691+0000","last_clean":"2024-03-11T18:30:32.844691+0000","last_became_active":"2024-03-11T18:30:32.317304+0000","last_became_peered":"2024-03-11T18:30:32.317304+0000","last_unstale":"2024-03-11T18:30:32.844691+0000","last_undegraded":"2024-03-11T18:30:32.844691+0000","last_fullsized":"2024-03-11T18:30:32.844691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:01.678144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.457547+0000","last_change":"2024-03-11T18:30:27.268370+0000","last_active":"2024-03-11T18:31:00.457547+0000","last_peered":"2024-03-11T18:31:00.457547+0000","last_clean":"2024-03-11T18:31:00.457547+0000","last_became_active":"2024-03-11T18:30:27.268033+0000","last_became_peered":"2024-03-11T18:30:27.268033+0000","last_unstale":"2024-03-11T18:31:00.457547+0000","last_undegraded":"2024-03-11T18:31:00.457547+0000","last_fullsized":"2024-03-11T18:31:00.457547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:31:13.545893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055113+0000","last_change":"2024-03-11T18:30:28.051611+0000","last_active":"2024-03-11T18:30:34.055113+0000","last_peered":"2024-03-11T18:30:34.055113+0000","last_clean":"2024-03-11T18:30:34.055113+0000","last_became_active":"2024-03-11T18:30:28.051249+0000","last_became_peered":"2024-03-11T18:30:28.051249+0000","last_unstale":"2024-03-11T18:30:34.055113+0000","last_undegraded":"2024-03-11T18:30:34.055113+0000","last_fullsized":"2024-03-11T18:30:34.055113+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:12.942910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844496+0000","last_change":"2024-03-11T18:30:32.318934+0000","last_active":"2024-03-11T18:30:32.844496+0000","last_peered":"2024-03-11T18:30:32.844496+0000","last_clean":"2024-03-11T18:30:32.844496+0000","last_became_active":"2024-03-11T18:30:32.318474+0000","last_became_peered":"2024-03-11T18:30:32.318474+0000","last_unstale":"2024-03-11T18:30:32.844496+0000","last_undegraded":"2024-03-11T18:30:32.844496+0000","last_fullsized":"2024-03-11T18:30:32.844496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:48:33.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055702+0000","last_change":"2024-03-11T18:30:27.268675+0000","last_active":"2024-03-11T18:30:34.055702+0000","last_peered":"2024-03-11T18:30:34.055702+0000","last_clean":"2024-03-11T18:30:34.055702+0000","last_became_active":"2024-03-11T18:30:27.268320+0000","last_became_peered":"2024-03-11T18:30:27.268320+0000","last_unstale":"2024-03-11T18:30:34.055702+0000","last_undegraded":"2024-03-11T18:30:34.055702+0000","last_fullsized":"2024-03-11T18:30:34.055702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:06.439930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056278+0000","last_change":"2024-03-11T18:30:28.038380+0000","last_active":"2024-03-11T18:30:34.056278+0000","last_peered":"2024-03-11T18:30:34.056278+0000","last_clean":"2024-03-11T18:30:34.056278+0000","last_became_active":"2024-03-11T18:30:28.037878+0000","last_became_peered":"2024-03-11T18:30:28.037878+0000","last_unstale":"2024-03-11T18:30:34.056278+0000","last_undegraded":"2024-03-11T18:30:34.056278+0000","last_fullsized":"2024-03-11T18:30:34.056278+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:10.988295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842311+0000","last_change":"2024-03-11T18:30:32.304747+0000","last_active":"2024-03-11T18:30:32.842311+0000","last_peered":"2024-03-11T18:30:32.842311+0000","last_clean":"2024-03-11T18:30:32.842311+0000","last_became_active":"2024-03-11T18:30:32.304282+0000","last_became_peered":"2024-03-11T18:30:32.304282+0000","last_unstale":"2024-03-11T18:30:32.842311+0000","last_undegraded":"2024-03-11T18:30:32.842311+0000","last_fullsized":"2024-03-11T18:30:32.842311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:42.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432710+0000","last_change":"2024-03-11T18:30:27.267574+0000","last_active":"2024-03-11T18:30:59.432710+0000","last_peered":"2024-03-11T18:30:59.432710+0000","last_clean":"2024-03-11T18:30:59.432710+0000","last_became_active":"2024-03-11T18:30:27.267278+0000","last_became_peered":"2024-03-11T18:30:27.267278+0000","last_unstale":"2024-03-11T18:30:59.432710+0000","last_undegraded":"2024-03-11T18:30:59.432710+0000","last_fullsized":"2024-03-11T18:30:59.432710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:28.997557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055488+0000","last_change":"2024-03-11T18:30:28.060280+0000","last_active":"2024-03-11T18:30:34.055488+0000","last_peered":"2024-03-11T18:30:34.055488+0000","last_clean":"2024-03-11T18:30:34.055488+0000","last_became_active":"2024-03-11T18:30:28.059929+0000","last_became_peered":"2024-03-11T18:30:28.059929+0000","last_unstale":"2024-03-11T18:30:34.055488+0000","last_undegraded":"2024-03-11T18:30:34.055488+0000","last_fullsized":"2024-03-11T18:30:34.055488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:47:31.995021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842034+0000","last_change":"2024-03-11T18:30:32.321754+0000","last_active":"2024-03-11T18:30:32.842034+0000","last_peered":"2024-03-11T18:30:32.842034+0000","last_clean":"2024-03-11T18:30:32.842034+0000","last_became_active":"2024-03-11T18:30:32.321329+0000","last_became_peered":"2024-03-11T18:30:32.321329+0000","last_unstale":"2024-03-11T18:30:32.842034+0000","last_undegraded":"2024-03-11T18:30:32.842034+0000","last_fullsized":"2024-03-11T18:30:32.842034+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:48.985005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432007+0000","last_change":"2024-03-11T18:30:27.285442+0000","last_active":"2024-03-11T18:30:33.432007+0000","last_peered":"2024-03-11T18:30:33.432007+0000","last_clean":"2024-03-11T18:30:33.432007+0000","last_became_active":"2024-03-11T18:30:27.285127+0000","last_became_peered":"2024-03-11T18:30:27.285127+0000","last_unstale":"2024-03-11T18:30:33.432007+0000","last_undegraded":"2024-03-11T18:30:33.432007+0000","last_fullsized":"2024-03-11T18:30:33.432007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:01.278846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056379+0000","last_change":"2024-03-11T18:30:28.036105+0000","last_active":"2024-03-11T18:30:34.056379+0000","last_peered":"2024-03-11T18:30:34.056379+0000","last_clean":"2024-03-11T18:30:34.056379+0000","last_became_active":"2024-03-11T18:30:28.035713+0000","last_became_peered":"2024-03-11T18:30:28.035713+0000","last_unstale":"2024-03-11T18:30:34.056379+0000","last_undegraded":"2024-03-11T18:30:34.056379+0000","last_fullsized":"2024-03-11T18:30:34.056379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:23:59.363573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057419+0000","last_change":"2024-03-11T18:30:32.327126+0000","last_active":"2024-03-11T18:30:34.057419+0000","last_peered":"2024-03-11T18:30:34.057419+0000","last_clean":"2024-03-11T18:30:34.057419+0000","last_became_active":"2024-03-11T18:30:32.326668+0000","last_became_peered":"2024-03-11T18:30:32.326668+0000","last_unstale":"2024-03-11T18:30:34.057419+0000","last_undegraded":"2024-03-11T18:30:34.057419+0000","last_fullsized":"2024-03-11T18:30:34.057419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:03.231714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465167+0000","last_change":"2024-03-11T18:30:27.289439+0000","last_active":"2024-03-11T18:31:00.465167+0000","last_peered":"2024-03-11T18:31:00.465167+0000","last_clean":"2024-03-11T18:31:00.465167+0000","last_became_active":"2024-03-11T18:30:27.286067+0000","last_became_peered":"2024-03-11T18:30:27.286067+0000","last_unstale":"2024-03-11T18:31:00.465167+0000","last_undegraded":"2024-03-11T18:31:00.465167+0000","last_fullsized":"2024-03-11T18:31:00.465167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:35.522841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059494+0000","last_change":"2024-03-11T18:30:28.058525+0000","last_active":"2024-03-11T18:30:34.059494+0000","last_peered":"2024-03-11T18:30:34.059494+0000","last_clean":"2024-03-11T18:30:34.059494+0000","last_became_active":"2024-03-11T18:30:28.058122+0000","last_became_peered":"2024-03-11T18:30:28.058122+0000","last_unstale":"2024-03-11T18:30:34.059494+0000","last_undegraded":"2024-03-11T18:30:34.059494+0000","last_fullsized":"2024-03-11T18:30:34.059494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:45:04.537880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843998+0000","last_change":"2024-03-11T18:30:32.314757+0000","last_active":"2024-03-11T18:30:32.843998+0000","last_peered":"2024-03-11T18:30:32.843998+0000","last_clean":"2024-03-11T18:30:32.843998+0000","last_became_active":"2024-03-11T18:30:32.313461+0000","last_became_peered":"2024-03-11T18:30:32.313461+0000","last_unstale":"2024-03-11T18:30:32.843998+0000","last_undegraded":"2024-03-11T18:30:32.843998+0000","last_fullsized":"2024-03-11T18:30:32.843998+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:11:27.881788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055764+0000","last_change":"2024-03-11T18:30:27.268359+0000","last_active":"2024-03-11T18:30:34.055764+0000","last_peered":"2024-03-11T18:30:34.055764+0000","last_clean":"2024-03-11T18:30:34.055764+0000","last_became_active":"2024-03-11T18:30:27.268045+0000","last_became_peered":"2024-03-11T18:30:27.268045+0000","last_unstale":"2024-03-11T18:30:34.055764+0000","last_undegraded":"2024-03-11T18:30:34.055764+0000","last_fullsized":"2024-03-11T18:30:34.055764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:53.678477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056587+0000","last_change":"2024-03-11T18:30:28.051331+0000","last_active":"2024-03-11T18:30:34.056587+0000","last_peered":"2024-03-11T18:30:34.056587+0000","last_clean":"2024-03-11T18:30:34.056587+0000","last_became_active":"2024-03-11T18:30:28.050462+0000","last_became_peered":"2024-03-11T18:30:28.050462+0000","last_unstale":"2024-03-11T18:30:34.056587+0000","last_undegraded":"2024-03-11T18:30:34.056587+0000","last_fullsized":"2024-03-11T18:30:34.056587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:24.194650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060286+0000","last_change":"2024-03-11T18:30:32.317173+0000","last_active":"2024-03-11T18:30:34.060286+0000","last_peered":"2024-03-11T18:30:34.060286+0000","last_clean":"2024-03-11T18:30:34.060286+0000","last_became_active":"2024-03-11T18:30:32.316721+0000","last_became_peered":"2024-03-11T18:30:32.316721+0000","last_unstale":"2024-03-11T18:30:34.060286+0000","last_undegraded":"2024-03-11T18:30:34.060286+0000","last_fullsized":"2024-03-11T18:30:34.060286+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:43.665949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432951+0000","last_change":"2024-03-11T18:30:27.279586+0000","last_active":"2024-03-11T18:30:59.432951+0000","last_peered":"2024-03-11T18:30:59.432951+0000","last_clean":"2024-03-11T18:30:59.432951+0000","last_became_active":"2024-03-11T18:30:27.279262+0000","last_became_peered":"2024-03-11T18:30:27.279262+0000","last_unstale":"2024-03-11T18:30:59.432951+0000","last_undegraded":"2024-03-11T18:30:59.432951+0000","last_fullsized":"2024-03-11T18:30:59.432951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:02:33.570755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055465+0000","last_change":"2024-03-11T18:30:28.060636+0000","last_active":"2024-03-11T18:30:34.055465+0000","last_peered":"2024-03-11T18:30:34.055465+0000","last_clean":"2024-03-11T18:30:34.055465+0000","last_became_active":"2024-03-11T18:30:28.060282+0000","last_became_peered":"2024-03-11T18:30:28.060282+0000","last_unstale":"2024-03-11T18:30:34.055465+0000","last_undegraded":"2024-03-11T18:30:34.055465+0000","last_fullsized":"2024-03-11T18:30:34.055465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:56.618563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056345+0000","last_change":"2024-03-11T18:30:32.314680+0000","last_active":"2024-03-11T18:30:34.056345+0000","last_peered":"2024-03-11T18:30:34.056345+0000","last_clean":"2024-03-11T18:30:34.056345+0000","last_became_active":"2024-03-11T18:30:32.314102+0000","last_became_peered":"2024-03-11T18:30:32.314102+0000","last_unstale":"2024-03-11T18:30:34.056345+0000","last_undegraded":"2024-03-11T18:30:34.056345+0000","last_fullsized":"2024-03-11T18:30:34.056345+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:18:55.989746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.431275+0000","last_change":"2024-03-11T18:30:27.267028+0000","last_active":"2024-03-11T18:30:59.431275+0000","last_peered":"2024-03-11T18:30:59.431275+0000","last_clean":"2024-03-11T18:30:59.431275+0000","last_became_active":"2024-03-11T18:30:27.266712+0000","last_became_peered":"2024-03-11T18:30:27.266712+0000","last_unstale":"2024-03-11T18:30:59.431275+0000","last_undegraded":"2024-03-11T18:30:59.431275+0000","last_fullsized":"2024-03-11T18:30:59.431275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:35.978113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844365+0000","last_change":"2024-03-11T18:30:28.066607+0000","last_active":"2024-03-11T18:30:32.844365+0000","last_peered":"2024-03-11T18:30:32.844365+0000","last_clean":"2024-03-11T18:30:32.844365+0000","last_became_active":"2024-03-11T18:30:28.061181+0000","last_became_peered":"2024-03-11T18:30:28.061181+0000","last_unstale":"2024-03-11T18:30:32.844365+0000","last_undegraded":"2024-03-11T18:30:32.844365+0000","last_fullsized":"2024-03-11T18:30:32.844365+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:01.203632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844947+0000","last_change":"2024-03-11T18:30:32.317424+0000","last_active":"2024-03-11T18:30:32.844947+0000","last_peered":"2024-03-11T18:30:32.844947+0000","last_clean":"2024-03-11T18:30:32.844947+0000","last_became_active":"2024-03-11T18:30:32.316633+0000","last_became_peered":"2024-03-11T18:30:32.316633+0000","last_unstale":"2024-03-11T18:30:32.844947+0000","last_undegraded":"2024-03-11T18:30:32.844947+0000","last_fullsized":"2024-03-11T18:30:32.844947+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:28:47.017770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056241+0000","last_change":"2024-03-11T18:30:27.270147+0000","last_active":"2024-03-11T18:30:34.056241+0000","last_peered":"2024-03-11T18:30:34.056241+0000","last_clean":"2024-03-11T18:30:34.056241+0000","last_became_active":"2024-03-11T18:30:27.269791+0000","last_became_peered":"2024-03-11T18:30:27.269791+0000","last_unstale":"2024-03-11T18:30:34.056241+0000","last_undegraded":"2024-03-11T18:30:34.056241+0000","last_fullsized":"2024-03-11T18:30:34.056241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:22.007518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055186+0000","last_change":"2024-03-11T18:30:28.035357+0000","last_active":"2024-03-11T18:30:34.055186+0000","last_peered":"2024-03-11T18:30:34.055186+0000","last_clean":"2024-03-11T18:30:34.055186+0000","last_became_active":"2024-03-11T18:30:28.035005+0000","last_became_peered":"2024-03-11T18:30:28.035005+0000","last_unstale":"2024-03-11T18:30:34.055186+0000","last_undegraded":"2024-03-11T18:30:34.055186+0000","last_fullsized":"2024-03-11T18:30:34.055186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:21.450440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056750+0000","last_change":"2024-03-11T18:30:32.323472+0000","last_active":"2024-03-11T18:30:34.056750+0000","last_peered":"2024-03-11T18:30:34.056750+0000","last_clean":"2024-03-11T18:30:34.056750+0000","last_became_active":"2024-03-11T18:30:32.321590+0000","last_became_peered":"2024-03-11T18:30:32.321590+0000","last_unstale":"2024-03-11T18:30:34.056750+0000","last_undegraded":"2024-03-11T18:30:34.056750+0000","last_fullsized":"2024-03-11T18:30:34.056750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:12.329127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.430384+0000","last_change":"2024-03-11T18:30:27.267683+0000","last_active":"2024-03-11T18:30:59.430384+0000","last_peered":"2024-03-11T18:30:59.430384+0000","last_clean":"2024-03-11T18:30:59.430384+0000","last_became_active":"2024-03-11T18:30:27.267333+0000","last_became_peered":"2024-03-11T18:30:27.267333+0000","last_unstale":"2024-03-11T18:30:59.430384+0000","last_undegraded":"2024-03-11T18:30:59.430384+0000","last_fullsized":"2024-03-11T18:30:59.430384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:33:18.721843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056403+0000","last_change":"2024-03-11T18:30:28.038846+0000","last_active":"2024-03-11T18:30:34.056403+0000","last_peered":"2024-03-11T18:30:34.056403+0000","last_clean":"2024-03-11T18:30:34.056403+0000","last_became_active":"2024-03-11T18:30:28.038479+0000","last_became_peered":"2024-03-11T18:30:28.038479+0000","last_unstale":"2024-03-11T18:30:34.056403+0000","last_undegraded":"2024-03-11T18:30:34.056403+0000","last_fullsized":"2024-03-11T18:30:34.056403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:25.877627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056767+0000","last_change":"2024-03-11T18:30:32.317761+0000","last_active":"2024-03-11T18:30:34.056767+0000","last_peered":"2024-03-11T18:30:34.056767+0000","last_clean":"2024-03-11T18:30:34.056767+0000","last_became_active":"2024-03-11T18:30:32.317438+0000","last_became_peered":"2024-03-11T18:30:32.317438+0000","last_unstale":"2024-03-11T18:30:34.056767+0000","last_undegraded":"2024-03-11T18:30:34.056767+0000","last_fullsized":"2024-03-11T18:30:34.056767+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:58.145037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.456980+0000","last_change":"2024-03-11T18:30:27.276641+0000","last_active":"2024-03-11T18:31:00.456980+0000","last_peered":"2024-03-11T18:31:00.456980+0000","last_clean":"2024-03-11T18:31:00.456980+0000","last_became_active":"2024-03-11T18:30:27.276293+0000","last_became_peered":"2024-03-11T18:30:27.276293+0000","last_unstale":"2024-03-11T18:31:00.456980+0000","last_undegraded":"2024-03-11T18:31:00.456980+0000","last_fullsized":"2024-03-11T18:31:00.456980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:46.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059448+0000","last_change":"2024-03-11T18:30:28.064304+0000","last_active":"2024-03-11T18:30:34.059448+0000","last_peered":"2024-03-11T18:30:34.059448+0000","last_clean":"2024-03-11T18:30:34.059448+0000","last_became_active":"2024-03-11T18:30:28.063923+0000","last_became_peered":"2024-03-11T18:30:28.063923+0000","last_unstale":"2024-03-11T18:30:34.059448+0000","last_undegraded":"2024-03-11T18:30:34.059448+0000","last_fullsized":"2024-03-11T18:30:34.059448+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:57:41.289717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057170+0000","last_change":"2024-03-11T18:30:32.325409+0000","last_active":"2024-03-11T18:30:34.057170+0000","last_peered":"2024-03-11T18:30:34.057170+0000","last_clean":"2024-03-11T18:30:34.057170+0000","last_became_active":"2024-03-11T18:30:32.324953+0000","last_became_peered":"2024-03-11T18:30:32.324953+0000","last_unstale":"2024-03-11T18:30:34.057170+0000","last_undegraded":"2024-03-11T18:30:34.057170+0000","last_fullsized":"2024-03-11T18:30:34.057170+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:44:54.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.434395+0000","last_change":"2024-03-11T18:30:27.271690+0000","last_active":"2024-03-11T18:30:59.434395+0000","last_peered":"2024-03-11T18:30:59.434395+0000","last_clean":"2024-03-11T18:30:59.434395+0000","last_became_active":"2024-03-11T18:30:27.271429+0000","last_became_peered":"2024-03-11T18:30:27.271429+0000","last_unstale":"2024-03-11T18:30:59.434395+0000","last_undegraded":"2024-03-11T18:30:59.434395+0000","last_fullsized":"2024-03-11T18:30:59.434395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:55:23.896642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962826+0000","last_change":"2024-03-11T18:30:28.054214+0000","last_active":"2024-03-11T18:30:33.962826+0000","last_peered":"2024-03-11T18:30:33.962826+0000","last_clean":"2024-03-11T18:30:33.962826+0000","last_became_active":"2024-03-11T18:30:28.053447+0000","last_became_peered":"2024-03-11T18:30:28.053447+0000","last_unstale":"2024-03-11T18:30:33.962826+0000","last_undegraded":"2024-03-11T18:30:33.962826+0000","last_fullsized":"2024-03-11T18:30:33.962826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:08:49.866509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058378+0000","last_change":"2024-03-11T18:30:32.325028+0000","last_active":"2024-03-11T18:30:34.058378+0000","last_peered":"2024-03-11T18:30:34.058378+0000","last_clean":"2024-03-11T18:30:34.058378+0000","last_became_active":"2024-03-11T18:30:32.324526+0000","last_became_peered":"2024-03-11T18:30:32.324526+0000","last_unstale":"2024-03-11T18:30:34.058378+0000","last_undegraded":"2024-03-11T18:30:34.058378+0000","last_fullsized":"2024-03-11T18:30:34.058378+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:08.031224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:02.473089+0000","last_change":"2024-03-11T18:30:27.265284+0000","last_active":"2024-03-11T18:31:02.473089+0000","last_peered":"2024-03-11T18:31:02.473089+0000","last_clean":"2024-03-11T18:31:02.473089+0000","last_became_active":"2024-03-11T18:30:27.264402+0000","last_became_peered":"2024-03-11T18:30:27.264402+0000","last_unstale":"2024-03-11T18:31:02.473089+0000","last_undegraded":"2024-03-11T18:31:02.473089+0000","last_fullsized":"2024-03-11T18:31:02.473089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:14.768956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962294+0000","last_change":"2024-03-11T18:30:28.040540+0000","last_active":"2024-03-11T18:30:33.962294+0000","last_peered":"2024-03-11T18:30:33.962294+0000","last_clean":"2024-03-11T18:30:33.962294+0000","last_became_active":"2024-03-11T18:30:28.040181+0000","last_became_peered":"2024-03-11T18:30:28.040181+0000","last_unstale":"2024-03-11T18:30:33.962294+0000","last_undegraded":"2024-03-11T18:30:33.962294+0000","last_fullsized":"2024-03-11T18:30:33.962294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:19.079250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061243+0000","last_change":"2024-03-11T18:30:32.326182+0000","last_active":"2024-03-11T18:30:34.061243+0000","last_peered":"2024-03-11T18:30:34.061243+0000","last_clean":"2024-03-11T18:30:34.061243+0000","last_became_active":"2024-03-11T18:30:32.325793+0000","last_became_peered":"2024-03-11T18:30:32.325793+0000","last_unstale":"2024-03-11T18:30:34.061243+0000","last_undegraded":"2024-03-11T18:30:34.061243+0000","last_fullsized":"2024-03-11T18:30:34.061243+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:14:13.212511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465352+0000","last_change":"2024-03-11T18:30:27.266724+0000","last_active":"2024-03-11T18:31:00.465352+0000","last_peered":"2024-03-11T18:31:00.465352+0000","last_clean":"2024-03-11T18:31:00.465352+0000","last_became_active":"2024-03-11T18:30:27.266394+0000","last_became_peered":"2024-03-11T18:30:27.266394+0000","last_unstale":"2024-03-11T18:31:00.465352+0000","last_undegraded":"2024-03-11T18:31:00.465352+0000","last_fullsized":"2024-03-11T18:31:00.465352+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:24.115722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056087+0000","last_change":"2024-03-11T18:30:28.039476+0000","last_active":"2024-03-11T18:30:34.056087+0000","last_peered":"2024-03-11T18:30:34.056087+0000","last_clean":"2024-03-11T18:30:34.056087+0000","last_became_active":"2024-03-11T18:30:28.038897+0000","last_became_peered":"2024-03-11T18:30:28.038897+0000","last_unstale":"2024-03-11T18:30:34.056087+0000","last_undegraded":"2024-03-11T18:30:34.056087+0000","last_fullsized":"2024-03-11T18:30:34.056087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:36:16.103173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056012+0000","last_change":"2024-03-11T18:30:32.319516+0000","last_active":"2024-03-11T18:30:34.056012+0000","last_peered":"2024-03-11T18:30:34.056012+0000","last_clean":"2024-03-11T18:30:34.056012+0000","last_became_active":"2024-03-11T18:30:32.319117+0000","last_became_peered":"2024-03-11T18:30:32.319117+0000","last_unstale":"2024-03-11T18:30:34.056012+0000","last_undegraded":"2024-03-11T18:30:34.056012+0000","last_fullsized":"2024-03-11T18:30:34.056012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:39.797872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:01.453787+0000","last_change":"2024-03-11T18:30:27.267543+0000","last_active":"2024-03-11T18:31:01.453787+0000","last_peered":"2024-03-11T18:31:01.453787+0000","last_clean":"2024-03-11T18:31:01.453787+0000","last_became_active":"2024-03-11T18:30:27.267235+0000","last_became_peered":"2024-03-11T18:30:27.267235+0000","last_unstale":"2024-03-11T18:31:01.453787+0000","last_undegraded":"2024-03-11T18:31:01.453787+0000","last_fullsized":"2024-03-11T18:31:01.453787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:24.556896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057284+0000","last_change":"2024-03-11T18:30:28.039401+0000","last_active":"2024-03-11T18:30:34.057284+0000","last_peered":"2024-03-11T18:30:34.057284+0000","last_clean":"2024-03-11T18:30:34.057284+0000","last_became_active":"2024-03-11T18:30:28.039054+0000","last_became_peered":"2024-03-11T18:30:28.039054+0000","last_unstale":"2024-03-11T18:30:34.057284+0000","last_undegraded":"2024-03-11T18:30:34.057284+0000","last_fullsized":"2024-03-11T18:30:34.057284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:01.701304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055532+0000","last_change":"2024-03-11T18:30:32.527930+0000","last_active":"2024-03-11T18:30:34.055532+0000","last_peered":"2024-03-11T18:30:34.055532+0000","last_clean":"2024-03-11T18:30:34.055532+0000","last_became_active":"2024-03-11T18:30:32.527371+0000","last_became_peered":"2024-03-11T18:30:32.527371+0000","last_unstale":"2024-03-11T18:30:34.055532+0000","last_undegraded":"2024-03-11T18:30:34.055532+0000","last_fullsized":"2024-03-11T18:30:34.055532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:09:52.361924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057270+0000","last_change":"2024-03-11T18:30:27.281098+0000","last_active":"2024-03-11T18:30:34.057270+0000","last_peered":"2024-03-11T18:30:34.057270+0000","last_clean":"2024-03-11T18:30:34.057270+0000","last_became_active":"2024-03-11T18:30:27.280796+0000","last_became_peered":"2024-03-11T18:30:27.280796+0000","last_unstale":"2024-03-11T18:30:34.057270+0000","last_undegraded":"2024-03-11T18:30:34.057270+0000","last_fullsized":"2024-03-11T18:30:34.057270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:28.661626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432320+0000","last_change":"2024-03-11T18:30:28.054079+0000","last_active":"2024-03-11T18:30:33.432320+0000","last_peered":"2024-03-11T18:30:33.432320+0000","last_clean":"2024-03-11T18:30:33.432320+0000","last_became_active":"2024-03-11T18:30:28.053715+0000","last_became_peered":"2024-03-11T18:30:28.053715+0000","last_unstale":"2024-03-11T18:30:33.432320+0000","last_undegraded":"2024-03-11T18:30:33.432320+0000","last_fullsized":"2024-03-11T18:30:33.432320+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:05.211602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.064381+0000","last_change":"2024-03-11T18:30:32.311132+0000","last_active":"2024-03-11T18:30:34.064381+0000","last_peered":"2024-03-11T18:30:34.064381+0000","last_clean":"2024-03-11T18:30:34.064381+0000","last_became_active":"2024-03-11T18:30:32.309585+0000","last_became_peered":"2024-03-11T18:30:32.309585+0000","last_unstale":"2024-03-11T18:30:34.064381+0000","last_undegraded":"2024-03-11T18:30:34.064381+0000","last_fullsized":"2024-03-11T18:30:34.064381+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:57.272479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962723+0000","last_change":"2024-03-11T18:30:27.288080+0000","last_active":"2024-03-11T18:30:33.962723+0000","last_peered":"2024-03-11T18:30:33.962723+0000","last_clean":"2024-03-11T18:30:33.962723+0000","last_became_active":"2024-03-11T18:30:27.287776+0000","last_became_peered":"2024-03-11T18:30:27.287776+0000","last_unstale":"2024-03-11T18:30:33.962723+0000","last_undegraded":"2024-03-11T18:30:33.962723+0000","last_fullsized":"2024-03-11T18:30:33.962723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:22.081753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.432210+0000","last_change":"2024-03-11T18:30:28.062011+0000","last_active":"2024-03-11T18:30:33.432210+0000","last_peered":"2024-03-11T18:30:33.432210+0000","last_clean":"2024-03-11T18:30:33.432210+0000","last_became_active":"2024-03-11T18:30:28.061592+0000","last_became_peered":"2024-03-11T18:30:28.061592+0000","last_unstale":"2024-03-11T18:30:33.432210+0000","last_undegraded":"2024-03-11T18:30:33.432210+0000","last_fullsized":"2024-03-11T18:30:33.432210+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:10.013980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061225+0000","last_change":"2024-03-11T18:30:32.528123+0000","last_active":"2024-03-11T18:30:34.061225+0000","last_peered":"2024-03-11T18:30:34.061225+0000","last_clean":"2024-03-11T18:30:34.061225+0000","last_became_active":"2024-03-11T18:30:32.527611+0000","last_became_peered":"2024-03-11T18:30:32.527611+0000","last_unstale":"2024-03-11T18:30:34.061225+0000","last_undegraded":"2024-03-11T18:30:34.061225+0000","last_fullsized":"2024-03-11T18:30:34.061225+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:28.501667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055953+0000","last_change":"2024-03-11T18:30:28.036930+0000","last_active":"2024-03-11T18:30:34.055953+0000","last_peered":"2024-03-11T18:30:34.055953+0000","last_clean":"2024-03-11T18:30:34.055953+0000","last_became_active":"2024-03-11T18:30:28.036286+0000","last_became_peered":"2024-03-11T18:30:28.036286+0000","last_unstale":"2024-03-11T18:30:34.055953+0000","last_undegraded":"2024-03-11T18:30:34.055953+0000","last_fullsized":"2024-03-11T18:30:34.055953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:10:40.239284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.468123+0000","last_change":"2024-03-11T18:30:27.287838+0000","last_active":"2024-03-11T18:31:00.468123+0000","last_peered":"2024-03-11T18:31:00.468123+0000","last_clean":"2024-03-11T18:31:00.468123+0000","last_became_active":"2024-03-11T18:30:27.287447+0000","last_became_peered":"2024-03-11T18:30:27.287447+0000","last_unstale":"2024-03-11T18:31:00.468123+0000","last_undegraded":"2024-03-11T18:31:00.468123+0000","last_fullsized":"2024-03-11T18:31:00.468123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:47.242977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963057+0000","last_change":"2024-03-11T18:30:32.321871+0000","last_active":"2024-03-11T18:30:33.963057+0000","last_peered":"2024-03-11T18:30:33.963057+0000","last_clean":"2024-03-11T18:30:33.963057+0000","last_became_active":"2024-03-11T18:30:32.321447+0000","last_became_peered":"2024-03-11T18:30:32.321447+0000","last_unstale":"2024-03-11T18:30:33.963057+0000","last_undegraded":"2024-03-11T18:30:33.963057+0000","last_fullsized":"2024-03-11T18:30:33.963057+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:24:56.124415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060833+0000","last_change":"2024-03-11T18:30:28.037906+0000","last_active":"2024-03-11T18:30:34.060833+0000","last_peered":"2024-03-11T18:30:34.060833+0000","last_clean":"2024-03-11T18:30:34.060833+0000","last_became_active":"2024-03-11T18:30:28.034143+0000","last_became_peered":"2024-03-11T18:30:28.034143+0000","last_unstale":"2024-03-11T18:30:34.060833+0000","last_undegraded":"2024-03-11T18:30:34.060833+0000","last_fullsized":"2024-03-11T18:30:34.060833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:36.728143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:01.463425+0000","last_change":"2024-03-11T18:30:27.273520+0000","last_active":"2024-03-11T18:31:01.463425+0000","last_peered":"2024-03-11T18:31:01.463425+0000","last_clean":"2024-03-11T18:31:01.463425+0000","last_became_active":"2024-03-11T18:30:27.273141+0000","last_became_peered":"2024-03-11T18:30:27.273141+0000","last_unstale":"2024-03-11T18:31:01.463425+0000","last_undegraded":"2024-03-11T18:31:01.463425+0000","last_fullsized":"2024-03-11T18:31:01.463425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:51.467579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962253+0000","last_change":"2024-03-11T18:29:08.539862+0000","last_active":"2024-03-11T18:30:33.962253+0000","last_peered":"2024-03-11T18:30:33.962253+0000","last_clean":"2024-03-11T18:30:33.962253+0000","last_became_active":"2024-03-11T18:29:08.539403+0000","last_became_peered":"2024-03-11T18:29:08.539403+0000","last_unstale":"2024-03-11T18:30:33.962253+0000","last_undegraded":"2024-03-11T18:30:33.962253+0000","last_fullsized":"2024-03-11T18:30:33.962253+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_clean_scrub_stamp":"2024-03-11T18:25:13.035017+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:24.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057109+0000","last_change":"2024-03-11T18:30:32.528841+0000","last_active":"2024-03-11T18:30:34.057109+0000","last_peered":"2024-03-11T18:30:34.057109+0000","last_clean":"2024-03-11T18:30:34.057109+0000","last_became_active":"2024-03-11T18:30:32.528320+0000","last_became_peered":"2024-03-11T18:30:32.528320+0000","last_unstale":"2024-03-11T18:30:34.057109+0000","last_undegraded":"2024-03-11T18:30:34.057109+0000","last_fullsized":"2024-03-11T18:30:34.057109+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:26.354490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058730+0000","last_change":"2024-03-11T18:30:28.047552+0000","last_active":"2024-03-11T18:30:34.058730+0000","last_peered":"2024-03-11T18:30:34.058730+0000","last_clean":"2024-03-11T18:30:34.058730+0000","last_became_active":"2024-03-11T18:30:28.046470+0000","last_became_peered":"2024-03-11T18:30:28.046470+0000","last_unstale":"2024-03-11T18:30:34.058730+0000","last_undegraded":"2024-03-11T18:30:34.058730+0000","last_fullsized":"2024-03-11T18:30:34.058730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:58:23.080406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.458046+0000","last_change":"2024-03-11T18:30:27.269416+0000","last_active":"2024-03-11T18:31:00.458046+0000","last_peered":"2024-03-11T18:31:00.458046+0000","last_clean":"2024-03-11T18:31:00.458046+0000","last_became_active":"2024-03-11T18:30:27.268951+0000","last_became_peered":"2024-03-11T18:30:27.268951+0000","last_unstale":"2024-03-11T18:31:00.458046+0000","last_undegraded":"2024-03-11T18:31:00.458046+0000","last_fullsized":"2024-03-11T18:31:00.458046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:03:16.138585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057439+0000","last_change":"2024-03-11T18:30:32.309247+0000","last_active":"2024-03-11T18:30:34.057439+0000","last_peered":"2024-03-11T18:30:34.057439+0000","last_clean":"2024-03-11T18:30:34.057439+0000","last_became_active":"2024-03-11T18:30:32.308761+0000","last_became_peered":"2024-03-11T18:30:32.308761+0000","last_unstale":"2024-03-11T18:30:34.057439+0000","last_undegraded":"2024-03-11T18:30:34.057439+0000","last_fullsized":"2024-03-11T18:30:34.057439+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:10.525154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.436989+0000","last_change":"2024-03-11T18:30:27.279543+0000","last_active":"2024-03-11T18:30:59.436989+0000","last_peered":"2024-03-11T18:30:59.436989+0000","last_clean":"2024-03-11T18:30:59.436989+0000","last_became_active":"2024-03-11T18:30:27.279203+0000","last_became_peered":"2024-03-11T18:30:27.279203+0000","last_unstale":"2024-03-11T18:30:59.436989+0000","last_undegraded":"2024-03-11T18:30:59.436989+0000","last_fullsized":"2024-03-11T18:30:59.436989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:31.792757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.064060+0000","last_change":"2024-03-11T18:30:28.060508+0000","last_active":"2024-03-11T18:30:34.064060+0000","last_peered":"2024-03-11T18:30:34.064060+0000","last_clean":"2024-03-11T18:30:34.064060+0000","last_became_active":"2024-03-11T18:30:28.059074+0000","last_became_peered":"2024-03-11T18:30:28.059074+0000","last_unstale":"2024-03-11T18:30:34.064060+0000","last_undegraded":"2024-03-11T18:30:34.064060+0000","last_fullsized":"2024-03-11T18:30:34.064060+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:34.204704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059426+0000","last_change":"2024-03-11T18:30:32.528017+0000","last_active":"2024-03-11T18:30:34.059426+0000","last_peered":"2024-03-11T18:30:34.059426+0000","last_clean":"2024-03-11T18:30:34.059426+0000","last_became_active":"2024-03-11T18:30:32.527518+0000","last_became_peered":"2024-03-11T18:30:32.527518+0000","last_unstale":"2024-03-11T18:30:34.059426+0000","last_undegraded":"2024-03-11T18:30:34.059426+0000","last_fullsized":"2024-03-11T18:30:34.059426+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:13:09.095889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.465943+0000","last_change":"2024-03-11T18:30:27.275540+0000","last_active":"2024-03-11T18:31:00.465943+0000","last_peered":"2024-03-11T18:31:00.465943+0000","last_clean":"2024-03-11T18:31:00.465943+0000","last_became_active":"2024-03-11T18:30:27.275070+0000","last_became_peered":"2024-03-11T18:30:27.275070+0000","last_unstale":"2024-03-11T18:31:00.465943+0000","last_undegraded":"2024-03-11T18:31:00.465943+0000","last_fullsized":"2024-03-11T18:31:00.465943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:59.729542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063246+0000","last_change":"2024-03-11T18:30:28.045096+0000","last_active":"2024-03-11T18:30:34.063246+0000","last_peered":"2024-03-11T18:30:34.063246+0000","last_clean":"2024-03-11T18:30:34.063246+0000","last_became_active":"2024-03-11T18:30:28.044408+0000","last_became_peered":"2024-03-11T18:30:28.044408+0000","last_unstale":"2024-03-11T18:30:34.063246+0000","last_undegraded":"2024-03-11T18:30:34.063246+0000","last_fullsized":"2024-03-11T18:30:34.063246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:49:59.426247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842052+0000","last_change":"2024-03-11T18:30:32.303288+0000","last_active":"2024-03-11T18:30:32.842052+0000","last_peered":"2024-03-11T18:30:32.842052+0000","last_clean":"2024-03-11T18:30:32.842052+0000","last_became_active":"2024-03-11T18:30:32.302746+0000","last_became_peered":"2024-03-11T18:30:32.302746+0000","last_unstale":"2024-03-11T18:30:32.842052+0000","last_undegraded":"2024-03-11T18:30:32.842052+0000","last_fullsized":"2024-03-11T18:30:32.842052+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:24.541448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.434794+0000","last_change":"2024-03-11T18:30:27.272880+0000","last_active":"2024-03-11T18:30:59.434794+0000","last_peered":"2024-03-11T18:30:59.434794+0000","last_clean":"2024-03-11T18:30:59.434794+0000","last_became_active":"2024-03-11T18:30:27.272611+0000","last_became_peered":"2024-03-11T18:30:27.272611+0000","last_unstale":"2024-03-11T18:30:59.434794+0000","last_undegraded":"2024-03-11T18:30:59.434794+0000","last_fullsized":"2024-03-11T18:30:59.434794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:41:10.322990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844385+0000","last_change":"2024-03-11T18:30:28.065613+0000","last_active":"2024-03-11T18:30:32.844385+0000","last_peered":"2024-03-11T18:30:32.844385+0000","last_clean":"2024-03-11T18:30:32.844385+0000","last_became_active":"2024-03-11T18:30:28.060520+0000","last_became_peered":"2024-03-11T18:30:28.060520+0000","last_unstale":"2024-03-11T18:30:32.844385+0000","last_undegraded":"2024-03-11T18:30:32.844385+0000","last_fullsized":"2024-03-11T18:30:32.844385+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:44.530168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845179+0000","last_change":"2024-03-11T18:30:32.528633+0000","last_active":"2024-03-11T18:30:32.845179+0000","last_peered":"2024-03-11T18:30:32.845179+0000","last_clean":"2024-03-11T18:30:32.845179+0000","last_became_active":"2024-03-11T18:30:32.528073+0000","last_became_peered":"2024-03-11T18:30:32.528073+0000","last_unstale":"2024-03-11T18:30:32.845179+0000","last_undegraded":"2024-03-11T18:30:32.845179+0000","last_fullsized":"2024-03-11T18:30:32.845179+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:38.942153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432059+0000","last_change":"2024-03-11T18:30:27.261435+0000","last_active":"2024-03-11T18:30:59.432059+0000","last_peered":"2024-03-11T18:30:59.432059+0000","last_clean":"2024-03-11T18:30:59.432059+0000","last_became_active":"2024-03-11T18:30:27.260070+0000","last_became_peered":"2024-03-11T18:30:27.260070+0000","last_unstale":"2024-03-11T18:30:59.432059+0000","last_undegraded":"2024-03-11T18:30:59.432059+0000","last_fullsized":"2024-03-11T18:30:59.432059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:11.785818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055234+0000","last_change":"2024-03-11T18:30:28.041708+0000","last_active":"2024-03-11T18:30:34.055234+0000","last_peered":"2024-03-11T18:30:34.055234+0000","last_clean":"2024-03-11T18:30:34.055234+0000","last_became_active":"2024-03-11T18:30:28.039647+0000","last_became_peered":"2024-03-11T18:30:28.039647+0000","last_unstale":"2024-03-11T18:30:34.055234+0000","last_undegraded":"2024-03-11T18:30:34.055234+0000","last_fullsized":"2024-03-11T18:30:34.055234+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.624175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.058652+0000","last_change":"2024-03-11T18:30:32.326512+0000","last_active":"2024-03-11T18:30:34.058652+0000","last_peered":"2024-03-11T18:30:34.058652+0000","last_clean":"2024-03-11T18:30:34.058652+0000","last_became_active":"2024-03-11T18:30:32.325253+0000","last_became_peered":"2024-03-11T18:30:32.325253+0000","last_unstale":"2024-03-11T18:30:34.058652+0000","last_undegraded":"2024-03-11T18:30:34.058652+0000","last_fullsized":"2024-03-11T18:30:34.058652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:48.589769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.435979+0000","last_change":"2024-03-11T18:30:27.286194+0000","last_active":"2024-03-11T18:30:59.435979+0000","last_peered":"2024-03-11T18:30:59.435979+0000","last_clean":"2024-03-11T18:30:59.435979+0000","last_became_active":"2024-03-11T18:30:27.284877+0000","last_became_peered":"2024-03-11T18:30:27.284877+0000","last_unstale":"2024-03-11T18:30:59.435979+0000","last_undegraded":"2024-03-11T18:30:59.435979+0000","last_fullsized":"2024-03-11T18:30:59.435979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:56:20.324266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056625+0000","last_change":"2024-03-11T18:30:28.061038+0000","last_active":"2024-03-11T18:30:34.056625+0000","last_peered":"2024-03-11T18:30:34.056625+0000","last_clean":"2024-03-11T18:30:34.056625+0000","last_became_active":"2024-03-11T18:30:28.060701+0000","last_became_peered":"2024-03-11T18:30:28.060701+0000","last_unstale":"2024-03-11T18:30:34.056625+0000","last_undegraded":"2024-03-11T18:30:34.056625+0000","last_fullsized":"2024-03-11T18:30:34.056625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:11.838309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.845215+0000","last_change":"2024-03-11T18:30:32.316762+0000","last_active":"2024-03-11T18:30:32.845215+0000","last_peered":"2024-03-11T18:30:32.845215+0000","last_clean":"2024-03-11T18:30:32.845215+0000","last_became_active":"2024-03-11T18:30:32.316318+0000","last_became_peered":"2024-03-11T18:30:32.316318+0000","last_unstale":"2024-03-11T18:30:32.845215+0000","last_undegraded":"2024-03-11T18:30:32.845215+0000","last_fullsized":"2024-03-11T18:30:32.845215+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:29.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.841658+0000","last_change":"2024-03-11T18:30:27.276636+0000","last_active":"2024-03-11T18:30:32.841658+0000","last_peered":"2024-03-11T18:30:32.841658+0000","last_clean":"2024-03-11T18:30:32.841658+0000","last_became_active":"2024-03-11T18:30:27.276290+0000","last_became_peered":"2024-03-11T18:30:27.276290+0000","last_unstale":"2024-03-11T18:30:32.841658+0000","last_undegraded":"2024-03-11T18:30:32.841658+0000","last_fullsized":"2024-03-11T18:30:32.841658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:34.272568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.062534+0000","last_change":"2024-03-11T18:30:28.067822+0000","last_active":"2024-03-11T18:30:34.062534+0000","last_peered":"2024-03-11T18:30:34.062534+0000","last_clean":"2024-03-11T18:30:34.062534+0000","last_became_active":"2024-03-11T18:30:28.067379+0000","last_became_peered":"2024-03-11T18:30:28.067379+0000","last_unstale":"2024-03-11T18:30:34.062534+0000","last_undegraded":"2024-03-11T18:30:34.062534+0000","last_fullsized":"2024-03-11T18:30:34.062534+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:27.730186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.844905+0000","last_change":"2024-03-11T18:30:32.527845+0000","last_active":"2024-03-11T18:30:32.844905+0000","last_peered":"2024-03-11T18:30:32.844905+0000","last_clean":"2024-03-11T18:30:32.844905+0000","last_became_active":"2024-03-11T18:30:32.527356+0000","last_became_peered":"2024-03-11T18:30:32.527356+0000","last_unstale":"2024-03-11T18:30:32.844905+0000","last_undegraded":"2024-03-11T18:30:32.844905+0000","last_fullsized":"2024-03-11T18:30:32.844905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.005118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:59.432713+0000","last_change":"2024-03-11T18:30:27.267263+0000","last_active":"2024-03-11T18:30:59.432713+0000","last_peered":"2024-03-11T18:30:59.432713+0000","last_clean":"2024-03-11T18:30:59.432713+0000","last_became_active":"2024-03-11T18:30:27.266893+0000","last_became_peered":"2024-03-11T18:30:27.266893+0000","last_unstale":"2024-03-11T18:30:59.432713+0000","last_undegraded":"2024-03-11T18:30:59.432713+0000","last_fullsized":"2024-03-11T18:30:59.432713+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:04.305623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962566+0000","last_change":"2024-03-11T18:30:28.054211+0000","last_active":"2024-03-11T18:30:33.962566+0000","last_peered":"2024-03-11T18:30:33.962566+0000","last_clean":"2024-03-11T18:30:33.962566+0000","last_became_active":"2024-03-11T18:30:28.053472+0000","last_became_peered":"2024-03-11T18:30:28.053472+0000","last_unstale":"2024-03-11T18:30:33.962566+0000","last_undegraded":"2024-03-11T18:30:33.962566+0000","last_fullsized":"2024-03-11T18:30:33.962566+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:37.415224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.054931+0000","last_change":"2024-03-11T18:30:32.304049+0000","last_active":"2024-03-11T18:30:34.054931+0000","last_peered":"2024-03-11T18:30:34.054931+0000","last_clean":"2024-03-11T18:30:34.054931+0000","last_became_active":"2024-03-11T18:30:32.303410+0000","last_became_peered":"2024-03-11T18:30:32.303410+0000","last_unstale":"2024-03-11T18:30:34.054931+0000","last_undegraded":"2024-03-11T18:30:34.054931+0000","last_fullsized":"2024-03-11T18:30:34.054931+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:15.574274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:31:00.458118+0000","last_change":"2024-03-11T18:30:27.263269+0000","last_active":"2024-03-11T18:31:00.458118+0000","last_peered":"2024-03-11T18:31:00.458118+0000","last_clean":"2024-03-11T18:31:00.458118+0000","last_became_active":"2024-03-11T18:30:27.262892+0000","last_became_peered":"2024-03-11T18:30:27.262892+0000","last_unstale":"2024-03-11T18:31:00.458118+0000","last_undegraded":"2024-03-11T18:31:00.458118+0000","last_fullsized":"2024-03-11T18:31:00.458118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:59.753276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.963108+0000","last_change":"2024-03-11T18:30:28.043362+0000","last_active":"2024-03-11T18:30:33.963108+0000","last_peered":"2024-03-11T18:30:33.963108+0000","last_clean":"2024-03-11T18:30:33.963108+0000","last_became_active":"2024-03-11T18:30:28.042566+0000","last_became_peered":"2024-03-11T18:30:28.042566+0000","last_unstale":"2024-03-11T18:30:33.963108+0000","last_undegraded":"2024-03-11T18:30:33.963108+0000","last_fullsized":"2024-03-11T18:30:33.963108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:41.226647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843110+0000","last_change":"2024-03-11T18:30:32.323886+0000","last_active":"2024-03-11T18:30:32.843110+0000","last_peered":"2024-03-11T18:30:32.843110+0000","last_clean":"2024-03-11T18:30:32.843110+0000","last_became_active":"2024-03-11T18:30:32.321785+0000","last_became_peered":"2024-03-11T18:30:32.321785+0000","last_unstale":"2024-03-11T18:30:32.843110+0000","last_undegraded":"2024-03-11T18:30:32.843110+0000","last_fullsized":"2024-03-11T18:30:32.843110+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:37:25.411440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056308+0000","last_change":"2024-03-11T18:30:27.271415+0000","last_active":"2024-03-11T18:30:34.056308+0000","last_peered":"2024-03-11T18:30:34.056308+0000","last_clean":"2024-03-11T18:30:34.056308+0000","last_became_active":"2024-03-11T18:30:27.271041+0000","last_became_peered":"2024-03-11T18:30:27.271041+0000","last_unstale":"2024-03-11T18:30:34.056308+0000","last_undegraded":"2024-03-11T18:30:34.056308+0000","last_fullsized":"2024-03-11T18:30:34.056308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:06.967984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962881+0000","last_change":"2024-03-11T18:30:28.042918+0000","last_active":"2024-03-11T18:30:33.962881+0000","last_peered":"2024-03-11T18:30:33.962881+0000","last_clean":"2024-03-11T18:30:33.962881+0000","last_became_active":"2024-03-11T18:30:28.042493+0000","last_became_peered":"2024-03-11T18:30:28.042493+0000","last_unstale":"2024-03-11T18:30:33.962881+0000","last_undegraded":"2024-03-11T18:30:33.962881+0000","last_fullsized":"2024-03-11T18:30:33.962881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:32:57.501997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055541+0000","last_change":"2024-03-11T18:30:32.304111+0000","last_active":"2024-03-11T18:30:34.055541+0000","last_peered":"2024-03-11T18:30:34.055541+0000","last_clean":"2024-03-11T18:30:34.055541+0000","last_became_active":"2024-03-11T18:30:32.303420+0000","last_became_peered":"2024-03-11T18:30:32.303420+0000","last_unstale":"2024-03-11T18:30:34.055541+0000","last_undegraded":"2024-03-11T18:30:34.055541+0000","last_fullsized":"2024-03-11T18:30:34.055541+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:51.801772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.061534+0000","last_change":"2024-03-11T18:30:27.288015+0000","last_active":"2024-03-11T18:30:34.061534+0000","last_peered":"2024-03-11T18:30:34.061534+0000","last_clean":"2024-03-11T18:30:34.061534+0000","last_became_active":"2024-03-11T18:30:27.287528+0000","last_became_peered":"2024-03-11T18:30:27.287528+0000","last_unstale":"2024-03-11T18:30:34.061534+0000","last_undegraded":"2024-03-11T18:30:34.061534+0000","last_fullsized":"2024-03-11T18:30:34.061534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:56:33.612120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:33.962910+0000","last_change":"2024-03-11T18:30:28.041888+0000","last_active":"2024-03-11T18:30:33.962910+0000","last_peered":"2024-03-11T18:30:33.962910+0000","last_clean":"2024-03-11T18:30:33.962910+0000","last_became_active":"2024-03-11T18:30:28.041506+0000","last_became_peered":"2024-03-11T18:30:28.041506+0000","last_unstale":"2024-03-11T18:30:33.962910+0000","last_undegraded":"2024-03-11T18:30:33.962910+0000","last_fullsized":"2024-03-11T18:30:33.962910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:46.308525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.056752+0000","last_change":"2024-03-11T18:30:28.037778+0000","last_active":"2024-03-11T18:30:34.056752+0000","last_peered":"2024-03-11T18:30:34.056752+0000","last_clean":"2024-03-11T18:30:34.056752+0000","last_became_active":"2024-03-11T18:30:28.037428+0000","last_became_peered":"2024-03-11T18:30:28.037428+0000","last_unstale":"2024-03-11T18:30:34.056752+0000","last_undegraded":"2024-03-11T18:30:34.056752+0000","last_fullsized":"2024-03-11T18:30:34.056752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:30:35.810425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842265+0000","last_change":"2024-03-11T18:30:28.037380+0000","last_active":"2024-03-11T18:30:32.842265+0000","last_peered":"2024-03-11T18:30:32.842265+0000","last_clean":"2024-03-11T18:30:32.842265+0000","last_became_active":"2024-03-11T18:30:28.037028+0000","last_became_peered":"2024-03-11T18:30:28.037028+0000","last_unstale":"2024-03-11T18:30:32.842265+0000","last_undegraded":"2024-03-11T18:30:32.842265+0000","last_fullsized":"2024-03-11T18:30:32.842265+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:34.161889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.841615+0000","last_change":"2024-03-11T18:30:28.060947+0000","last_active":"2024-03-11T18:30:32.841615+0000","last_peered":"2024-03-11T18:30:32.841615+0000","last_clean":"2024-03-11T18:30:32.841615+0000","last_became_active":"2024-03-11T18:30:28.060600+0000","last_became_peered":"2024-03-11T18:30:28.060600+0000","last_unstale":"2024-03-11T18:30:32.841615+0000","last_undegraded":"2024-03-11T18:30:32.841615+0000","last_fullsized":"2024-03-11T18:30:32.841615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:31.964347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.843351+0000","last_change":"2024-03-11T18:30:28.062110+0000","last_active":"2024-03-11T18:30:32.843351+0000","last_peered":"2024-03-11T18:30:32.843351+0000","last_clean":"2024-03-11T18:30:32.843351+0000","last_became_active":"2024-03-11T18:30:28.061427+0000","last_became_peered":"2024-03-11T18:30:28.061427+0000","last_unstale":"2024-03-11T18:30:32.843351+0000","last_undegraded":"2024-03-11T18:30:32.843351+0000","last_fullsized":"2024-03-11T18:30:32.843351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:32.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063886+0000","last_change":"2024-03-11T18:30:28.060118+0000","last_active":"2024-03-11T18:30:34.063886+0000","last_peered":"2024-03-11T18:30:34.063886+0000","last_clean":"2024-03-11T18:30:34.063886+0000","last_became_active":"2024-03-11T18:30:28.059733+0000","last_became_peered":"2024-03-11T18:30:28.059733+0000","last_unstale":"2024-03-11T18:30:34.063886+0000","last_undegraded":"2024-03-11T18:30:34.063886+0000","last_fullsized":"2024-03-11T18:30:34.063886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:18:56.059082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057307+0000","last_change":"2024-03-11T18:30:28.041817+0000","last_active":"2024-03-11T18:30:34.057307+0000","last_peered":"2024-03-11T18:30:34.057307+0000","last_clean":"2024-03-11T18:30:34.057307+0000","last_became_active":"2024-03-11T18:30:28.041346+0000","last_became_peered":"2024-03-11T18:30:28.041346+0000","last_unstale":"2024-03-11T18:30:34.057307+0000","last_undegraded":"2024-03-11T18:30:34.057307+0000","last_fullsized":"2024-03-11T18:30:34.057307+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:26:28.544602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:32.842271+0000","last_change":"2024-03-11T18:30:28.040820+0000","last_active":"2024-03-11T18:30:32.842271+0000","last_peered":"2024-03-11T18:30:32.842271+0000","last_clean":"2024-03-11T18:30:32.842271+0000","last_became_active":"2024-03-11T18:30:28.040381+0000","last_became_peered":"2024-03-11T18:30:28.040381+0000","last_unstale":"2024-03-11T18:30:32.842271+0000","last_undegraded":"2024-03-11T18:30:32.842271+0000","last_fullsized":"2024-03-11T18:30:32.842271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:33:37.120786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.063449+0000","last_change":"2024-03-11T18:30:28.052050+0000","last_active":"2024-03-11T18:30:34.063449+0000","last_peered":"2024-03-11T18:30:34.063449+0000","last_clean":"2024-03-11T18:30:34.063449+0000","last_became_active":"2024-03-11T18:30:28.051685+0000","last_became_peered":"2024-03-11T18:30:28.051685+0000","last_unstale":"2024-03-11T18:30:34.063449+0000","last_undegraded":"2024-03-11T18:30:34.063449+0000","last_fullsized":"2024-03-11T18:30:34.063449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:02.777603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055831+0000","last_change":"2024-03-11T18:30:28.053993+0000","last_active":"2024-03-11T18:30:34.055831+0000","last_peered":"2024-03-11T18:30:34.055831+0000","last_clean":"2024-03-11T18:30:34.055831+0000","last_became_active":"2024-03-11T18:30:28.053678+0000","last_became_peered":"2024-03-11T18:30:28.053678+0000","last_unstale":"2024-03-11T18:30:34.055831+0000","last_undegraded":"2024-03-11T18:30:34.055831+0000","last_fullsized":"2024-03-11T18:30:34.055831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:48.801078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.059246+0000","last_change":"2024-03-11T18:30:28.051357+0000","last_active":"2024-03-11T18:30:34.059246+0000","last_peered":"2024-03-11T18:30:34.059246+0000","last_clean":"2024-03-11T18:30:34.059246+0000","last_became_active":"2024-03-11T18:30:28.051046+0000","last_became_peered":"2024-03-11T18:30:28.051046+0000","last_unstale":"2024-03-11T18:30:34.059246+0000","last_undegraded":"2024-03-11T18:30:34.059246+0000","last_fullsized":"2024-03-11T18:30:34.059246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:11.994754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.055143+0000","last_change":"2024-03-11T18:30:28.051628+0000","last_active":"2024-03-11T18:30:34.055143+0000","last_peered":"2024-03-11T18:30:34.055143+0000","last_clean":"2024-03-11T18:30:34.055143+0000","last_became_active":"2024-03-11T18:30:28.051129+0000","last_became_peered":"2024-03-11T18:30:28.051129+0000","last_unstale":"2024-03-11T18:30:34.055143+0000","last_undegraded":"2024-03-11T18:30:34.055143+0000","last_fullsized":"2024-03-11T18:30:34.055143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:47.568747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.060554+0000","last_change":"2024-03-11T18:30:28.065792+0000","last_active":"2024-03-11T18:30:34.060554+0000","last_peered":"2024-03-11T18:30:34.060554+0000","last_clean":"2024-03-11T18:30:34.060554+0000","last_became_active":"2024-03-11T18:30:28.064902+0000","last_became_peered":"2024-03-11T18:30:28.064902+0000","last_unstale":"2024-03-11T18:30:34.060554+0000","last_undegraded":"2024-03-11T18:30:34.060554+0000","last_fullsized":"2024-03-11T18:30:34.060554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:18.019872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T18:30:34.057149+0000","last_change":"2024-03-11T18:30:28.041663+0000","last_active":"2024-03-11T18:30:34.057149+0000","last_peered":"2024-03-11T18:30:34.057149+0000","last_clean":"2024-03-11T18:30:34.057149+0000","last_became_active":"2024-03-11T18:30:28.041257+0000","last_became_peered":"2024-03-11T18:30:28.041257+0000","last_unstale":"2024-03-11T18:30:34.057149+0000","last_undegraded":"2024-03-11T18:30:34.057149+0000","last_fullsized":"2024-03-11T18:30:34.057149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:18.216295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7843,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":15,"num_read_kb":13,"num_write":126,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":28835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318291,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":483896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1630208,"data_stored":1079062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":484714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":484634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266731,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1609728,"data_stored":1076838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430256,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":483986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":487738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":484590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":489088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117057,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":484090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855420,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":484540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691894,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1075740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":10442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T18:31:05.980 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-11T18:31:05.980 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T18:31:05.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T18:31:06.500 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:06.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:06 smithi002 ceph-mon[31328]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:06.844 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:06 smithi002 ceph-mon[31328]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:06.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:06 smithi165 ceph-mon[36634]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:06.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:06 smithi165 ceph-mon[36634]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:06.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:06 smithi023 ceph-mon[36406]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:06.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:06 smithi023 ceph-mon[36406]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:31:07.806 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:07 smithi165 ceph-mon[36634]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T18:31:07.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:07 smithi023 ceph-mon[36406]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T18:31:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:07 smithi002 ceph-mon[31328]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 -- 172.21.15.2:0/1957846219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 msgr2=0x7fa2ac1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1957846219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac1020e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0099c0 tx=0x7fa29c009cd0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 -- 172.21.15.2:0/1957846219 shutdown_connections 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1957846219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac103860 0x7fa2ac10a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1957846219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac1020e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1957846219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 -- 172.21.15.2:0/1957846219 >> 172.21.15.2:0/1957846219 conn(0x7fa2ac0fd280 msgr2=0x7fa2ac0ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.870+0000 7fa2b3000700 1 -- 172.21.15.2:0/1957846219 shutdown_connections 2024-03-11T18:31:08.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.871+0000 7fa2b3000700 1 -- 172.21.15.2:0/1957846219 wait complete. 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 Processor -- start 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 -- start start 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 0x7fa2ac1a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac1a1970 con 0x7fa2ac101cc0 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac1a1ab0 con 0x7fa2ac103860 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.872+0000 7fa2b3000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2ac1a1c40 con 0x7fa2ac102ec0 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2b1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 0x7fa2ac1a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58278/0 (socket says 172.21.15.2:58278) 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2ab400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:08.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 -- 172.21.15.2:0/1678779782 learned_addr learned my addr 172.21.15.2:0/1678779782 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 -- 172.21.15.2:0/1678779782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 msgr2=0x7fa2ac19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 -- 172.21.15.2:0/1678779782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 msgr2=0x7fa2ac1a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 0x7fa2ac1a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2abe00700 1 -- 172.21.15.2:0/1678779782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0095f0 con 0x7fa2ac101cc0 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2ab400700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.873+0000 7fa2b1000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 0x7fa2ac1a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:08.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2abe00700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009990 tx=0x7fa29c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c01d070 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa29c022470 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c017650 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2ac1a1ec0 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.874+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2ac1a2270 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.875+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2ac050180 con 0x7fa2ac101cc0 2024-03-11T18:31:08.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.877+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fa29c0225e0 con 0x7fa2ac101cc0 2024-03-11T18:31:08.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.878+0000 7fa2a8c00700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 0x7fa2900a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:08.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.879+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa29c0ce860 con 0x7fa2ac101cc0 2024-03-11T18:31:08.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.879+0000 7fa2ab400700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 0x7fa2900a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:08.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.880+0000 7fa2ab400700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 0x7fa2900a4140 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa298011490 tx=0x7fa298009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:08.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:08.883+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa29c097700 con 0x7fa2ac101cc0 2024-03-11T18:31:09.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.194+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa2ac10d930 con 0x7fa2ac101cc0 2024-03-11T18:31:09.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.198+0000 7fa2a8c00700 1 -- 172.21.15.2:0/1678779782 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa2ac10d930 con 0x7fa2ac101cc0 2024-03-11T18:31:09.200 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-03-11T18:31:09.200 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDtTe9lWOemCxAASyHVUlWt1Q2jVUJJATRJMg== 2024-03-11T18:31:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 msgr2=0x7fa2900a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 0x7fa2900a4140 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa298011490 tx=0x7fa298009260 comp rx=0 tx=0).stop 2024-03-11T18:31:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 msgr2=0x7fa2ac199de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:09.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009990 tx=0x7fa29c00f690 comp rx=0 tx=0).stop 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 shutdown_connections 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2ac103860 0x7fa2ac1a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2ac101cc0 0x7fa2ac199de0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa2900a1c80 0x7fa2900a4140 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 --2- 172.21.15.2:0/1678779782 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2ac102ec0 0x7fa2ac19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 >> 172.21.15.2:0/1678779782 conn(0x7fa2ac0fd280 msgr2=0x7fa2ac073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:09.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 shutdown_connections 2024-03-11T18:31:09.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:09.202+0000 7fa2b3000700 1 -- 172.21.15.2:0/1678779782 wait complete. 2024-03-11T18:31:09.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:09 smithi002 ceph-mon[31328]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:09.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:09 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T18:31:09.767 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:09 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:09 smithi165 ceph-mon[36634]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:09 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T18:31:09.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:09 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T18:31:09.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:09 smithi023 ceph-mon[36406]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:09.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:09 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T18:31:09.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:09 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1678779782' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T18:31:10.144 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:31:10.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T18:31:10.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T18:31:10.310 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T18:31:10.310 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T18:31:10.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T18:31:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:11.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:11 smithi002 ceph-mon[31328]: pgmap v342: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-11T18:31:11.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:11 smithi165 ceph-mon[36634]: pgmap v342: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-11T18:31:11.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:11 smithi023 ceph-mon[36406]: pgmap v342: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-03-11T18:31:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.062+0000 7f5abc200700 1 -- 172.21.15.2:0/3934013916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab406a240 msgr2=0x7f5ab410e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.062+0000 7f5abc200700 1 --2- 172.21.15.2:0/3934013916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab406a240 0x7f5ab410e1e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009990 tx=0x7f5aa0009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 -- 172.21.15.2:0/3934013916 shutdown_connections 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 --2- 172.21.15.2:0/3934013916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4069920 0x7f5ab4069d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 --2- 172.21.15.2:0/3934013916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab410e720 0x7f5ab4110b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 --2- 172.21.15.2:0/3934013916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab406a240 0x7f5ab410e1e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 -- 172.21.15.2:0/3934013916 >> 172.21.15.2:0/3934013916 conn(0x7f5ab4077aa0 msgr2=0x7f5ab4077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 -- 172.21.15.2:0/3934013916 shutdown_connections 2024-03-11T18:31:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.063+0000 7f5abc200700 1 -- 172.21.15.2:0/3934013916 wait complete. 2024-03-11T18:31:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.065+0000 7f5abc200700 1 Processor -- start 2024-03-11T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.065+0000 7f5abc200700 1 -- start start 2024-03-11T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4069920 0x7f5ab419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab410e720 0x7f5ab419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab419f320 con 0x7f5ab410e720 2024-03-11T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab419f490 con 0x7f5ab4069920 2024-03-11T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5abc200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab419f650 con 0x7f5ab406a240 2024-03-11T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5ab8e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:13.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5ab8e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42234/0 (socket says 172.21.15.2:42234) 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5ab8e00700 1 -- 172.21.15.2:0/2067750073 learned_addr learned my addr 172.21.15.2:0/2067750073 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5aba200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab410e720 0x7f5ab419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.066+0000 7f5ab9800700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4069920 0x7f5ab419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 -- 172.21.15.2:0/2067750073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4069920 msgr2=0x7f5ab419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4069920 0x7f5ab419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 -- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab410e720 msgr2=0x7f5ab419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab410e720 0x7f5ab419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 -- 172.21.15.2:0/2067750073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac009c40 con 0x7f5ab406a240 2024-03-11T18:31:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab8e00700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0005910 tx=0x7f5aa0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa001d070 con 0x7f5ab406a240 2024-03-11T18:31:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.067+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa00095f0 con 0x7f5ab406a240 2024-03-11T18:31:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.068+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab41a6fc0 con 0x7f5ab406a240 2024-03-11T18:31:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.068+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa000b610 con 0x7f5ab406a240 2024-03-11T18:31:13.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.068+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00173f0 con 0x7f5ab406a240 2024-03-11T18:31:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.069+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4109940 con 0x7f5ab406a240 2024-03-11T18:31:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.071+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f5aa0022470 con 0x7f5ab406a240 2024-03-11T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.072+0000 7f5ab2000700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 0x7f5a9c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.072+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5aa00ce9b0 con 0x7f5ab406a240 2024-03-11T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.072+0000 7f5ab9800700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 0x7f5a9c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.073+0000 7f5ab9800700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 0x7f5a9c0a4090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009c60 tx=0x7f5aa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:13.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.075+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5aa0097850 con 0x7f5ab406a240 2024-03-11T18:31:13.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.359+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5ab419b800 con 0x7f5ab406a240 2024-03-11T18:31:13.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.366+0000 7f5ab2000700 1 -- 172.21.15.2:0/2067750073 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260653 (secure 0 0 0) 0x7f5aa0027090 con 0x7f5ab406a240 2024-03-11T18:31:13.369 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:13.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 msgr2=0x7f5a9c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:13.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 0x7f5a9c0a4090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8009c60 tx=0x7f5aa8009260 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 msgr2=0x7f5ab419a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0005910 tx=0x7f5aa0004730 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 shutdown_connections 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab410e720 0x7f5ab419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4069920 0x7f5ab419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1bd0 0x7f5a9c0a4090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 --2- 172.21.15.2:0/2067750073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab406a240 0x7f5ab419a980 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 >> 172.21.15.2:0/2067750073 conn(0x7f5ab4077aa0 msgr2=0x7f5ab410fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.372+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 shutdown_connections 2024-03-11T18:31:13.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:13.373+0000 7f5abc200700 1 -- 172.21.15.2:0/2067750073 wait complete. 2024-03-11T18:31:13.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:13 smithi002 ceph-mon[31328]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-03-11T18:31:13.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:13 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2067750073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T18:31:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:13 smithi023 ceph-mon[36406]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-03-11T18:31:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:13 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2067750073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T18:31:13.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:13 smithi165 ceph-mon[36634]: pgmap v343: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.4 KiB/s wr, 14 op/s 2024-03-11T18:31:13.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:13 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2067750073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T18:31:14.188 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":1446641624},{"type":"v1","addr":"172.21.15.2:6801","nonce":1446641624}]},"active_addr":"172.21.15.2:6801/1446641624","active_change":"2024-03-11T18:22:28.050502+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3571811795}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":492479102}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2218375343}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3512553041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2989450862}]}]} 2024-03-11T18:31:14.192 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T18:31:14.193 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T18:31:14.193 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T18:31:14.617 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:15.818 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:15 smithi002 ceph-mon[31328]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:15.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:15 smithi165 ceph-mon[36634]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:15.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:15 smithi023 ceph-mon[36406]: pgmap v344: 129 pgs: 129 active+clean; 626 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.743+0000 7f9a85600700 1 -- 172.21.15.2:0/476419582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80107d00 msgr2=0x7f9a801080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.743+0000 7f9a85600700 1 --2- 172.21.15.2:0/476419582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80107d00 0x7f9a801080e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009990 tx=0x7f9a70009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 -- 172.21.15.2:0/476419582 shutdown_connections 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 --2- 172.21.15.2:0/476419582 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80102690 0x7f9a8010eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 --2- 172.21.15.2:0/476419582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80107d00 0x7f9a801080e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 --2- 172.21.15.2:0/476419582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a80102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 -- 172.21.15.2:0/476419582 >> 172.21.15.2:0/476419582 conn(0x7f9a80078d40 msgr2=0x7f9a80079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:16.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 -- 172.21.15.2:0/476419582 shutdown_connections 2024-03-11T18:31:16.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.744+0000 7f9a85600700 1 -- 172.21.15.2:0/476419582 wait complete. 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 Processor -- start 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 -- start start 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a8019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 0x7f9a8019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80107d00 0x7f9a8019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8019f310 con 0x7f9a80101cf0 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8019f480 con 0x7f9a80102690 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.745+0000 7f9a85600700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8019f640 con 0x7f9a80107d00 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a7ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a8019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a7ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a8019a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58308/0 (socket says 172.21.15.2:58308) 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a7ea00700 1 -- 172.21.15.2:0/3190443336 learned_addr learned my addr 172.21.15.2:0/3190443336 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 0x7f9a8019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a7f400700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80107d00 0x7f9a8019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 -- 172.21.15.2:0/3190443336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80107d00 msgr2=0x7f9a8019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80107d00 0x7f9a8019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 -- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 msgr2=0x7f9a8019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a8019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.746+0000 7f9a77e00700 1 -- 172.21.15.2:0/3190443336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68009520 con 0x7f9a80102690 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a77e00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 0x7f9a8019a970 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800e9f0 tx=0x7f9a6800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6800cb70 con 0x7f9a80102690 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a6800ee90 con 0x7f9a80102690 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a700095f0 con 0x7f9a80102690 2024-03-11T18:31:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68003d80 con 0x7f9a80102690 2024-03-11T18:31:16.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.747+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a801a6fb0 con 0x7f9a80102690 2024-03-11T18:31:16.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.748+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a80068930 con 0x7f9a80102690 2024-03-11T18:31:16.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.750+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f9a68003ee0 con 0x7f9a80102690 2024-03-11T18:31:16.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.751+0000 7f9a77400700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 0x7f9a640a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:16.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.751+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9a68014070 con 0x7f9a80102690 2024-03-11T18:31:16.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.752+0000 7f9a7ea00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 0x7f9a640a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:16.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.752+0000 7f9a7ea00700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 0x7f9a640a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006010 tx=0x7f9a7000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.756+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9a68096880 con 0x7f9a80102690 2024-03-11T18:31:16.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.979+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a8019b790 con 0x7f9a80102690 2024-03-11T18:31:16.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.981+0000 7f9a77400700 1 -- 172.21.15.2:0/3190443336 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f9a68099f40 con 0x7f9a80102690 2024-03-11T18:31:16.983 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:16.984 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:31:16.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.984+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 msgr2=0x7f9a640a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.984+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 0x7f9a640a4090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9a70006010 tx=0x7f9a7000b260 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.984+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 msgr2=0x7f9a8019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.984+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 0x7f9a8019a970 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800e9f0 tx=0x7f9a6800ed00 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 shutdown_connections 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f9a640a1bd0 0x7f9a640a4090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a80107d00 0x7f9a8019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a80101cf0 0x7f9a8019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 --2- 172.21.15.2:0/3190443336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a80102690 0x7f9a8019a970 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 >> 172.21.15.2:0/3190443336 conn(0x7f9a80078d40 msgr2=0x7f9a8010d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:16.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 shutdown_connections 2024-03-11T18:31:16.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:16.985+0000 7f9a85600700 1 -- 172.21.15.2:0/3190443336 wait complete. 2024-03-11T18:31:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:17 smithi002 ceph-mon[31328]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-11T18:31:17.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3190443336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:17.901 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T18:31:17.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T18:31:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:17 smithi165 ceph-mon[36634]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-11T18:31:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3190443336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:17.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:17 smithi023 ceph-mon[36406]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-11T18:31:17.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3190443336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:18.374 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:19.556 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:19 smithi002 ceph-mon[31328]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:19.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:19 smithi023 ceph-mon[36406]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:19 smithi165 ceph-mon[36634]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T18:31:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 -- 172.21.15.2:0/3306919644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 msgr2=0x7f24741080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 --2- 172.21.15.2:0/3306919644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741080a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2464009990 tx=0x7f2464009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 -- 172.21.15.2:0/3306919644 shutdown_connections 2024-03-11T18:31:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 --2- 172.21.15.2:0/3306919644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f247410eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 --2- 172.21.15.2:0/3306919644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741080a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 --2- 172.21.15.2:0/3306919644 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 0x7f2474069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 -- 172.21.15.2:0/3306919644 >> 172.21.15.2:0/3306919644 conn(0x7f2474077aa0 msgr2=0x7f2474077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:20.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 -- 172.21.15.2:0/3306919644 shutdown_connections 2024-03-11T18:31:20.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.525+0000 7f2479e00700 1 -- 172.21.15.2:0/3306919644 wait complete. 2024-03-11T18:31:20.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.527+0000 7f2479e00700 1 Processor -- start 2024-03-11T18:31:20.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 -- start start 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 0x7f2474100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f2474101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741a65f0 con 0x7f247406a2c0 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741a6760 con 0x7f2474107cc0 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.528+0000 7f2479e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24741a6920 con 0x7f2474069920 2024-03-11T18:31:20.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f2472a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 0x7f2474100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:20.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f2473400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:20.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f246be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f2474101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:20.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f2473400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45892/0 (socket says 172.21.15.2:45892) 2024-03-11T18:31:20.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f2473400700 1 -- 172.21.15.2:0/3888853691 learned_addr learned my addr 172.21.15.2:0/3888853691 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:20.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f246be00700 1 -- 172.21.15.2:0/3888853691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 msgr2=0x7f2474100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.529+0000 7f246be00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 0x7f2474100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f246be00700 1 -- 172.21.15.2:0/3888853691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 msgr2=0x7f24741016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f246be00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f246be00700 1 -- 172.21.15.2:0/3888853691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c009c40 con 0x7f247406a2c0 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f2473400700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:31:20.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f246be00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f2474101180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f245c00e9f0 tx=0x7f245c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00eee0 con 0x7f247406a2c0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f245c00cbd0 con 0x7f247406a2c0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c018680 con 0x7f247406a2c0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.530+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640095f0 con 0x7f247406a2c0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.531+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24741a6f20 con 0x7f247406a2c0 2024-03-11T18:31:20.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.532+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247410a1e0 con 0x7f247406a2c0 2024-03-11T18:31:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.534+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f245c00cd40 con 0x7f247406a2c0 2024-03-11T18:31:20.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.536+0000 7f2478a00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 0x7f24580a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:20.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.536+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f245c014070 con 0x7f247406a2c0 2024-03-11T18:31:20.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.536+0000 7f2472a00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 0x7f24580a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:20.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.537+0000 7f2472a00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 0x7f24580a4090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f246400b2e0 tx=0x7f2464005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.541+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f245c052d20 con 0x7f247406a2c0 2024-03-11T18:31:20.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.772+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24741021d0 con 0x7f247406a2c0 2024-03-11T18:31:20.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.775+0000 7f2478a00700 1 -- 172.21.15.2:0/3888853691 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24047 (secure 0 0 0) 0x7f245c0967e0 con 0x7f247406a2c0 2024-03-11T18:31:20.778 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:31:20.779 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":91,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:30:32.770059+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:31:20.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 msgr2=0x7f24580a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 0x7f24580a4090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f246400b2e0 tx=0x7f2464005960 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 msgr2=0x7f2474101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f2474101180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f245c00e9f0 tx=0x7f245c00ed00 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 shutdown_connections 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f24580a1bd0 0x7f24580a4090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2474069920 0x7f2474100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2474107cc0 0x7f24741016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 --2- 172.21.15.2:0/3888853691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f247406a2c0 0x7f2474101180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.778+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 >> 172.21.15.2:0/3888853691 conn(0x7f2474077aa0 msgr2=0x7f24740ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.779+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 shutdown_connections 2024-03-11T18:31:20.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:20.779+0000 7f2479e00700 1 -- 172.21.15.2:0/3888853691 wait complete. 2024-03-11T18:31:21.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T18:31:21.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T18:31:21.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T18:31:21.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T18:31:21.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T18:31:21.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T18:31:21.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T18:31:21.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T18:31:21.611 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-11T18:31:21.611 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-11T18:31:21.612 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-11T18:31:21.612 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-11T18:31:21.786 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:21 smithi002 ceph-mon[31328]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-11T18:31:21.786 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:21 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3888853691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:21.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:21 smithi165 ceph-mon[36634]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-11T18:31:21.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3888853691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:21.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:21 smithi023 ceph-mon[36406]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-03-11T18:31:21.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:21 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3888853691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:31:22.712 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.806 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.824 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.825 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.835 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.837 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:22.837 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:23.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:23 smithi002 ceph-mon[31328]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T18:31:23.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:23 smithi165 ceph-mon[36634]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T18:31:23.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:23 smithi023 ceph-mon[36406]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-03-11T18:31:24.035 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:25.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:25 smithi002 ceph-mon[31328]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T18:31:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:25 smithi165 ceph-mon[36634]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T18:31:25.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:25 smithi023 ceph-mon[36406]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-11T18:31:27.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:27 smithi165 ceph-mon[36634]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T18:31:27.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:27 smithi023 ceph-mon[36406]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T18:31:28.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:27 smithi002 ceph-mon[31328]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-11T18:31:28.684 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.819 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T18:31:28.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]: dispatch 2024-03-11T18:31:28.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-11T18:31:28.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:31:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-11T18:31:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-11T18:31:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-11T18:31:29.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-11T18:31:29.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:29 smithi023 ceph-mon[36406]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-11T18:31:29.860 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:29 smithi002 ceph-mon[31328]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T18:31:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-11T18:31:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-03-11T18:31:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8, 5, 9]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 8]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-11T18:31:29.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:29 smithi165 ceph-mon[36634]: osdmap e92: 12 total, 12 up, 12 in 2024-03-11T18:31:30.499 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:30.502 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:30.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:30 smithi002 ceph-mon[31328]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T18:31:30.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:30 smithi023 ceph-mon[36406]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T18:31:30.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:30 smithi165 ceph-mon[36634]: osdmap e93: 12 total, 12 up, 12 in 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:30.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:31:32.219 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:32 smithi002 ceph-mon[31328]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-11T18:31:32.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:32 smithi023 ceph-mon[36406]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-11T18:31:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:32 smithi165 ceph-mon[36634]: pgmap v354: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 639 B/s wr, 6 op/s 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 -- 172.21.15.2:0/3906122713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 msgr2=0x7f4034110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 --2- 172.21.15.2:0/3906122713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 0x7f4034110c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f402c00b0a0 tx=0x7f402c00b3b0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 -- 172.21.15.2:0/3906122713 shutdown_connections 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 --2- 172.21.15.2:0/3906122713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f40341043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 --2- 172.21.15.2:0/3906122713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 0x7f4034110c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 --2- 172.21.15.2:0/3906122713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 0x7f40341089c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 -- 172.21.15.2:0/3906122713 >> 172.21.15.2:0/3906122713 conn(0x7f40340ff890 msgr2=0x7f4034101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 -- 172.21.15.2:0/3906122713 shutdown_connections 2024-03-11T18:31:33.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.040+0000 7f4038a00700 1 -- 172.21.15.2:0/3906122713 wait complete. 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.041+0000 7f4038a00700 1 Processor -- start 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 -- start start 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f403419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 0x7f403419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 0x7f403419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403419f220 con 0x7f40341090a0 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403419f390 con 0x7f4034104970 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4038a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403419f550 con 0x7f4034103fc0 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4032800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f403419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4031e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 0x7f403419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:33.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4032800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f403419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51334/0 (socket says 172.21.15.2:51334) 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4032800700 1 -- 172.21.15.2:0/1767134960 learned_addr learned my addr 172.21.15.2:0/1767134960 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.042+0000 7f4033200700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 0x7f403419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 -- 172.21.15.2:0/1767134960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 msgr2=0x7f403419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f403419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 -- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 msgr2=0x7f403419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40341090a0 0x7f403419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c009d00 con 0x7f4034104970 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4032800700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4034103fc0 0x7f403419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f4031e00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 0x7f403419a880 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f402000e9c0 tx=0x7f402000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402000c9c0 con 0x7f4034104970 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.043+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4020010430 con 0x7f4034104970 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.044+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403419f830 con 0x7f4034104970 2024-03-11T18:31:33.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.044+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020018640 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.044+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40341a7090 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.044+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4034114100 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.047+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f40200105a0 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f402b400700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f40180a1bd0 0x7f40180a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4020014070 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f4032800700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f40180a1bd0 0x7f40180a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f402b400700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] conn(0x7f40180accb0 0x7f40180af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40180af820 con 0x7f40180accb0 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.048+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40200ce330 con 0x7f4034104970 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.049+0000 7f4033200700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] conn(0x7f40180accb0 0x7f40180af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.049+0000 7f4032800700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f40180a1bd0 0x7f40180a4090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4024011550 tx=0x7f4024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.049+0000 7f4033200700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] conn(0x7f40180accb0 0x7f40180af150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:33.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.050+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== osd.3 v2:172.21.15.2:6810/2581962705 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40180af820 con 0x7f40180accb0 2024-03-11T18:31:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.069+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 --> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4034100c00 con 0x7f40180accb0 2024-03-11T18:31:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f402b400700 1 -- 172.21.15.2:0/1767134960 <== osd.3 v2:172.21.15.2:6810/2581962705 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4034100c00 con 0x7f40180accb0 2024-03-11T18:31:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] conn(0x7f40180accb0 msgr2=0x7f40180af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] conn(0x7f40180accb0 0x7f40180af150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f40180a1bd0 msgr2=0x7f40180a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f40180a1bd0 0x7f40180a4090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4024011550 tx=0x7f4024009260 comp rx=0 tx=0).stop 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 msgr2=0x7f403419a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 --2- 172.21.15.2:0/1767134960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4034104970 0x7f403419a880 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f402000e9c0 tx=0x7f402000ecd0 comp rx=0 tx=0).stop 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4033200700 1 -- 172.21.15.2:0/1767134960 reap_dead start 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 shutdown_connections 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.070+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 >> 172.21.15.2:0/1767134960 conn(0x7f40340ff890 msgr2=0x7f40341002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.071+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 shutdown_connections 2024-03-11T18:31:33.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:33.071+0000 7f4038a00700 1 -- 172.21.15.2:0/1767134960 wait complete. 2024-03-11T18:31:34.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 -- 172.21.15.2:0/3500186290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c44107cc0 msgr2=0x7f5c441080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 --2- 172.21.15.2:0/3500186290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c44107cc0 0x7f5c441080a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5c340098f0 tx=0x7f5c34009c00 comp rx=0 tx=0).stop 2024-03-11T18:31:34.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 -- 172.21.15.2:0/3500186290 shutdown_connections 2024-03-11T18:31:34.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 --2- 172.21.15.2:0/3500186290 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c4406a2c0 0x7f5c4410eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 --2- 172.21.15.2:0/3500186290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c44107cc0 0x7f5c441080a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 --2- 172.21.15.2:0/3500186290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c44069920 0x7f5c44069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 -- 172.21.15.2:0/3500186290 >> 172.21.15.2:0/3500186290 conn(0x7f5c44077aa0 msgr2=0x7f5c44077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 -- 172.21.15.2:0/3500186290 shutdown_connections 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.014+0000 7f5c49000700 1 -- 172.21.15.2:0/3500186290 wait complete. 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.015+0000 7f5c49000700 1 Processor -- start 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.015+0000 7f5c49000700 1 -- start start 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c44069920 0x7f5c44100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 0x7f5c44101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c44107cc0 0x7f5c441016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441a65a0 con 0x7f5c44107cc0 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441a6710 con 0x7f5c4406a2c0 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c49000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c441a68d0 con 0x7f5c44069920 2024-03-11T18:31:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 0x7f5c44101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c42800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c44107cc0 0x7f5c441016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 0x7f5c44101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37812/0 (socket says 172.21.15.2:37812) 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 -- 172.21.15.2:0/662290088 learned_addr learned my addr 172.21.15.2:0/662290088 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c41e00700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c44069920 0x7f5c44100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 -- 172.21.15.2:0/662290088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c44069920 msgr2=0x7f5c44100c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c44069920 0x7f5c44100c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 -- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c44107cc0 msgr2=0x7f5c441016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c44107cc0 0x7f5c441016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.016+0000 7f5c39400700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009520 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c39400700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 0x7f5c44101180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00e9c0 tx=0x7f5c2c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c004a30 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c441a6bb0 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c2c00cca0 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c005580 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.017+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c441a70b0 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.018+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5c44111f70 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.020+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f5c2c004c20 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c3b400700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5c280a1b50 0x7f5c280a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f5c2c014070 con 0x7f5c4406a2c0 2024-03-11T18:31:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c41e00700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5c280a1b50 0x7f5c280a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c3b400700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] conn(0x7f5c280acc30 0x7f5c280af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c280af7a0 con 0x7f5c280acc30 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.022+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c2c0ce000 con 0x7f5c4406a2c0 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.023+0000 7f5c42800700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] conn(0x7f5c280acc30 0x7f5c280af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.023+0000 7f5c41e00700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5c280a1b50 0x7f5c280a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009810 tx=0x7f5c34019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.023+0000 7f5c42800700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] conn(0x7f5c280acc30 0x7f5c280af0d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.024+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== osd.6 v2:172.21.15.2:6818/2663871562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5c280af7a0 con 0x7f5c280acc30 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.044+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 --> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5c44062380 con 0x7f5c280acc30 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.044+0000 7f5c3b400700 1 -- 172.21.15.2:0/662290088 <== osd.6 v2:172.21.15.2:6818/2663871562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5c44062380 con 0x7f5c280acc30 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] conn(0x7f5c280acc30 msgr2=0x7f5c280af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] conn(0x7f5c280acc30 0x7f5c280af0d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5c280a1b50 msgr2=0x7f5c280a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5c280a1b50 0x7f5c280a4010 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009810 tx=0x7f5c34019040 comp rx=0 tx=0).stop 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 msgr2=0x7f5c44101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.045+0000 7f5c49000700 1 --2- 172.21.15.2:0/662290088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c4406a2c0 0x7f5c44101180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00e9c0 tx=0x7f5c2c00ecd0 comp rx=0 tx=0).stop 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.046+0000 7f5c42800700 1 -- 172.21.15.2:0/662290088 reap_dead start 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.046+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 shutdown_connections 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.046+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 >> 172.21.15.2:0/662290088 conn(0x7f5c44077aa0 msgr2=0x7f5c440ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.046+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 shutdown_connections 2024-03-11T18:31:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:34.046+0000 7f5c49000700 1 -- 172.21.15.2:0/662290088 wait complete. 2024-03-11T18:31:34.170 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:34 smithi002 ceph-mon[31328]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-03-11T18:31:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:34 smithi165 ceph-mon[36634]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-03-11T18:31:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:34 smithi023 ceph-mon[36406]: pgmap v355: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 639 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-03-11T18:31:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 -- 172.21.15.2:0/1449701007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f80107cc0 msgr2=0x7f5f801080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 --2- 172.21.15.2:0/1449701007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f80107cc0 0x7f5f801080a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000b0d0 tx=0x7f5f7000b3e0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 -- 172.21.15.2:0/1449701007 shutdown_connections 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 --2- 172.21.15.2:0/1449701007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f8006a2c0 0x7f5f8010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 --2- 172.21.15.2:0/1449701007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f80107cc0 0x7f5f801080a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 --2- 172.21.15.2:0/1449701007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f80069920 0x7f5f80069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.406+0000 7f5f85a00700 1 -- 172.21.15.2:0/1449701007 >> 172.21.15.2:0/1449701007 conn(0x7f5f80077aa0 msgr2=0x7f5f80077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.407+0000 7f5f85a00700 1 -- 172.21.15.2:0/1449701007 shutdown_connections 2024-03-11T18:31:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.407+0000 7f5f85a00700 1 -- 172.21.15.2:0/1449701007 wait complete. 2024-03-11T18:31:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 Processor -- start 2024-03-11T18:31:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 -- start start 2024-03-11T18:31:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 0x7f5f80198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f8006a2c0 0x7f5f801987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f80107cc0 0x7f5f8019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8019d170 con 0x7f5f8006a2c0 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8019d2e0 con 0x7f5f80107cc0 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.408+0000 7f5f85a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8019d4a0 con 0x7f5f80069920 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f77e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f8006a2c0 0x7f5f801987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 0x7f5f80198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f77e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f8006a2c0 0x7f5f801987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33340/0 (socket says 172.21.15.2:33340) 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f77e00700 1 -- 172.21.15.2:0/4133145608 learned_addr learned my addr 172.21.15.2:0/4133145608 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 0x7f5f80198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51352/0 (socket says 172.21.15.2:51352) 2024-03-11T18:31:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7f400700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f80107cc0 0x7f5f8019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 -- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f80107cc0 msgr2=0x7f5f8019cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f80107cc0 0x7f5f8019cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 -- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f8006a2c0 msgr2=0x7f5f801987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f8006a2c0 0x7f5f801987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.409+0000 7f5f7ea00700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f70009d00 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f7ea00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 0x7f5f80198290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000f80 tx=0x7f5f7000bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000e040 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f700079d0 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7001d480 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8019d720 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.410+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f801a4e10 con 0x7f5f80069920 2024-03-11T18:31:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.411+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f80103370 con 0x7f5f80069920 2024-03-11T18:31:35.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.413+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f5f70005190 con 0x7f5f80069920 2024-03-11T18:31:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.414+0000 7f5f77400700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5f640a1bd0 0x7f5f640a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.414+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f5f700ce7d0 con 0x7f5f80069920 2024-03-11T18:31:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f77e00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5f640a1bd0 0x7f5f640a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f77400700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] conn(0x7f5f640accb0 0x7f5f640af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f640af820 con 0x7f5f640accb0 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f700ceb40 con 0x7f5f80069920 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f7f400700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] conn(0x7f5f640accb0 0x7f5f640af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.415+0000 7f5f77e00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5f640a1bd0 0x7f5f640a4090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f800780a0 tx=0x7f5f68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.416+0000 7f5f7f400700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] conn(0x7f5f640accb0 0x7f5f640af150 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.417+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== osd.1 v2:172.21.15.23:6800/1383496716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5f640af820 con 0x7f5f640accb0 2024-03-11T18:31:35.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.435+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 --> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f80111f90 con 0x7f5f640accb0 2024-03-11T18:31:35.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.436+0000 7f5f77400700 1 -- 172.21.15.2:0/4133145608 <== osd.1 v2:172.21.15.23:6800/1383496716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f80111f90 con 0x7f5f640accb0 2024-03-11T18:31:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] conn(0x7f5f640accb0 msgr2=0x7f5f640af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] conn(0x7f5f640accb0 0x7f5f640af150 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5f640a1bd0 msgr2=0x7f5f640a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5f640a1bd0 0x7f5f640a4090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f800780a0 tx=0x7f5f68009260 comp rx=0 tx=0).stop 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 msgr2=0x7f5f80198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 --2- 172.21.15.2:0/4133145608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f80069920 0x7f5f80198290 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000f80 tx=0x7f5f7000bd50 comp rx=0 tx=0).stop 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f7f400700 1 -- 172.21.15.2:0/4133145608 reap_dead start 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 shutdown_connections 2024-03-11T18:31:35.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 >> 172.21.15.2:0/4133145608 conn(0x7f5f80077aa0 msgr2=0x7f5f800ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 shutdown_connections 2024-03-11T18:31:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:35.437+0000 7f5f85a00700 1 -- 172.21.15.2:0/4133145608 wait complete. 2024-03-11T18:31:36.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:36 smithi002 ceph-mon[31328]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:36 smithi165 ceph-mon[36634]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:36.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:36 smithi023 ceph-mon[36406]: pgmap v356: 129 pgs: 129 active+clean; 641 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:38 smithi002 ceph-mon[31328]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:38 smithi165 ceph-mon[36634]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:38.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:38 smithi023 ceph-mon[36406]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 5 op/s; 141 B/s, 0 objects/s recovering 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 -- 172.21.15.2:0/2060590139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 msgr2=0x7f4d84074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2060590139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d84074420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009990 tx=0x7f4d74009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 -- 172.21.15.2:0/2060590139 shutdown_connections 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2060590139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d8410e530 0x7f4d84110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2060590139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d84074420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2060590139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d84074960 0x7f4d8410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 -- 172.21.15.2:0/2060590139 >> 172.21.15.2:0/2060590139 conn(0x7f4d840fd740 msgr2=0x7f4d840ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 -- 172.21.15.2:0/2060590139 shutdown_connections 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.136+0000 7f4d8b200700 1 -- 172.21.15.2:0/2060590139 wait complete. 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.137+0000 7f4d8b200700 1 Processor -- start 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.137+0000 7f4d8b200700 1 -- start start 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d8419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d84074960 0x7f4d8419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d8410e530 0x7f4d8419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419f270 con 0x7f4d84074040 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419f3e0 con 0x7f4d8410e530 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d8b200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419f5a0 con 0x7f4d84074960 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d83e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d8419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:39.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d83e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d8419a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49738/0 (socket says 172.21.15.2:49738) 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d83e00700 1 -- 172.21.15.2:0/2818634128 learned_addr learned my addr 172.21.15.2:0/2818634128 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d83400700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d84074960 0x7f4d8419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.138+0000 7f4d89200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d8410e530 0x7f4d8419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 -- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d84074960 msgr2=0x7f4d8419a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d84074960 0x7f4d8419a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 -- 172.21.15.2:0/2818634128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d8410e530 msgr2=0x7f4d8419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d8410e530 0x7f4d8419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d740095f0 con 0x7f4d84074040 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83400700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4d84074960 0x7f4d8419a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d83e00700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d8419a390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000c00 tx=0x7f4d74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7401d070 con 0x7f4d84074040 2024-03-11T18:31:39.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d74022470 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74017650 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.139+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8419f820 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.140+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d841a6f10 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.140+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d84114210 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.142+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f4d740225e0 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d80c00700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4d680a1bd0 0x7f4d680a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4d740cf320 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d83400700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4d680a1bd0 0x7f4d680a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d80c00700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] conn(0x7f4d680accb0 0x7f4d680af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d680af820 con 0x7f4d680accb0 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.144+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d740cf690 con 0x7f4d84074040 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.145+0000 7f4d89200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] conn(0x7f4d680accb0 0x7f4d680af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:39.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.145+0000 7f4d83400700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4d680a1bd0 0x7f4d680a4090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d8410fce0 tx=0x7f4d700073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.146+0000 7f4d89200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] conn(0x7f4d680accb0 0x7f4d680af150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.147+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== osd.8 v2:172.21.15.165:6816/2939503461 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4d680af820 con 0x7f4d680accb0 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.166+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 --> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4d84062380 con 0x7f4d680accb0 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d80c00700 1 -- 172.21.15.2:0/2818634128 <== osd.8 v2:172.21.15.165:6816/2939503461 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4d84062380 con 0x7f4d680accb0 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] conn(0x7f4d680accb0 msgr2=0x7f4d680af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] conn(0x7f4d680accb0 0x7f4d680af150 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4d680a1bd0 msgr2=0x7f4d680a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4d680a1bd0 0x7f4d680a4090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d8410fce0 tx=0x7f4d700073d0 comp rx=0 tx=0).stop 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 msgr2=0x7f4d8419a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d8b200700 1 --2- 172.21.15.2:0/2818634128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d84074040 0x7f4d8419a390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d74000c00 tx=0x7f4d74004730 comp rx=0 tx=0).stop 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.167+0000 7f4d89200700 1 -- 172.21.15.2:0/2818634128 reap_dead start 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.168+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 shutdown_connections 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.168+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 >> 172.21.15.2:0/2818634128 conn(0x7f4d840fd740 msgr2=0x7f4d840fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.168+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 shutdown_connections 2024-03-11T18:31:39.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:38.168+0000 7f4d8b200700 1 -- 172.21.15.2:0/2818634128 wait complete. 2024-03-11T18:31:40.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:40 smithi165 ceph-mon[36634]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 212 B/s wr, 5 op/s; 117 B/s, 0 objects/s recovering 2024-03-11T18:31:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:40 smithi023 ceph-mon[36406]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 212 B/s wr, 5 op/s; 117 B/s, 0 objects/s recovering 2024-03-11T18:31:40.485 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:40 smithi002 ceph-mon[31328]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 212 B/s wr, 5 op/s; 117 B/s, 0 objects/s recovering 2024-03-11T18:31:40.566 INFO:teuthology.orchestra.run.smithi002.stdout:115964117064 2024-03-11T18:31:40.567 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T18:31:41.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 -- 172.21.15.2:0/864601349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730100a60 msgr2=0x7fc730100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 --2- 172.21.15.2:0/864601349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730100a60 0x7fc730100ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009990 tx=0x7fc71c009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 -- 172.21.15.2:0/864601349 shutdown_connections 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 --2- 172.21.15.2:0/864601349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730101420 0x7fc73010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 --2- 172.21.15.2:0/864601349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730106a60 0x7fc730106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 --2- 172.21.15.2:0/864601349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730100a60 0x7fc730100ee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 -- 172.21.15.2:0/864601349 >> 172.21.15.2:0/864601349 conn(0x7fc7300fc550 msgr2=0x7fc7300fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 -- 172.21.15.2:0/864601349 shutdown_connections 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.565+0000 7fc736c00700 1 -- 172.21.15.2:0/864601349 wait complete. 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.567+0000 7fc736c00700 1 Processor -- start 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.567+0000 7fc736c00700 1 -- start start 2024-03-11T18:31:41.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.567+0000 7fc736c00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730100a60 0x7fc73010e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.567+0000 7fc736c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730101420 0x7fc73010e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.567+0000 7fc736c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730106a60 0x7fc73010ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc736c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301a6490 con 0x7fc730106a60 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc736c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301a6600 con 0x7fc730101420 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc736c00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301a67c0 con 0x7fc730100a60 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc735800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730100a60 0x7fc73010e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc735800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730100a60 0x7fc73010e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48384/0 (socket says 172.21.15.2:48384) 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc735800700 1 -- 172.21.15.2:0/173854855 learned_addr learned my addr 172.21.15.2:0/173854855 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730101420 0x7fc73010e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc736200700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730106a60 0x7fc73010ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 -- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730100a60 msgr2=0x7fc73010e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc730100a60 0x7fc73010e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 -- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730106a60 msgr2=0x7fc73010ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc730106a60 0x7fc73010ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.568+0000 7fc734e00700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009520 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc734e00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730101420 0x7fc73010e700 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc71c000c00 tx=0x7fc71c004710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c01d070 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c0095f0 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc71c00b610 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c021d80 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.569+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7301a6cc0 con 0x7fc730101420 2024-03-11T18:31:41.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.570+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc730111f30 con 0x7fc730101420 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.572+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fc71c021410 con 0x7fc730101420 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc72e000700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7180a1b50 0x7fc7180a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc71c0cf470 con 0x7fc730101420 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc735800700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7180a1b50 0x7fc7180a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc72e000700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] conn(0x7fc7180acc30 0x7fc7180af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7180af7a0 con 0x7fc7180acc30 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.574+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc71c0cf7e0 con 0x7fc730101420 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.575+0000 7fc736200700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] conn(0x7fc7180acc30 0x7fc7180af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.575+0000 7fc735800700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7180a1b50 0x7fc7180a4010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc724009e70 tx=0x7fc724009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.575+0000 7fc736200700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] conn(0x7fc7180acc30 0x7fc7180af0d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:41.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.576+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== osd.5 v2:172.21.15.165:6808/1264754008 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc7180af7a0 con 0x7fc7180acc30 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.595+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 --> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc730062380 con 0x7fc7180acc30 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.595+0000 7fc72e000700 1 -- 172.21.15.2:0/173854855 <== osd.5 v2:172.21.15.165:6808/1264754008 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc730062380 con 0x7fc7180acc30 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] conn(0x7fc7180acc30 msgr2=0x7fc7180af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] conn(0x7fc7180acc30 0x7fc7180af0d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7180a1b50 msgr2=0x7fc7180a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc7180a1b50 0x7fc7180a4010 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc724009e70 tx=0x7fc724009260 comp rx=0 tx=0).stop 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730101420 msgr2=0x7fc73010e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736c00700 1 --2- 172.21.15.2:0/173854855 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc730101420 0x7fc73010e700 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc71c000c00 tx=0x7fc71c004710 comp rx=0 tx=0).stop 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.596+0000 7fc736200700 1 -- 172.21.15.2:0/173854855 reap_dead start 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.597+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 shutdown_connections 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.597+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 >> 172.21.15.2:0/173854855 conn(0x7fc7300fc550 msgr2=0x7fc7300fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.597+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 shutdown_connections 2024-03-11T18:31:41.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:41.597+0000 7fc736c00700 1 -- 172.21.15.2:0/173854855 wait complete. 2024-03-11T18:31:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:42 smithi002 ceph-mon[31328]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 192 B/s wr, 4 op/s; 106 B/s, 0 objects/s recovering 2024-03-11T18:31:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:42 smithi165 ceph-mon[36634]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 192 B/s wr, 4 op/s; 106 B/s, 0 objects/s recovering 2024-03-11T18:31:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:42 smithi023 ceph-mon[36406]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 192 B/s wr, 4 op/s; 106 B/s, 0 objects/s recovering 2024-03-11T18:31:42.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 -- 172.21.15.2:0/800251219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c88108e90 msgr2=0x7f4c8810cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 --2- 172.21.15.2:0/800251219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c88108e90 0x7f4c8810cee0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009990 tx=0x7f4c74009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 -- 172.21.15.2:0/800251219 shutdown_connections 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 --2- 172.21.15.2:0/800251219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8810d5c0 0x7f4c88115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 --2- 172.21.15.2:0/800251219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c881084e0 0x7f4c881088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.819+0000 7f4c90400700 1 --2- 172.21.15.2:0/800251219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c88108e90 0x7f4c8810cee0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.820+0000 7f4c90400700 1 -- 172.21.15.2:0/800251219 >> 172.21.15.2:0/800251219 conn(0x7f4c88075440 msgr2=0x7f4c88077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.820+0000 7f4c90400700 1 -- 172.21.15.2:0/800251219 shutdown_connections 2024-03-11T18:31:42.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.820+0000 7f4c90400700 1 -- 172.21.15.2:0/800251219 wait complete. 2024-03-11T18:31:42.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 Processor -- start 2024-03-11T18:31:42.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 -- start start 2024-03-11T18:31:42.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c881084e0 0x7f4c881a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 0x7f4c881a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8810d5c0 0x7f4c8819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8819edf0 con 0x7f4c88108e90 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8819ef60 con 0x7f4c8810d5c0 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.821+0000 7f4c90400700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8819f120 con 0x7f4c881084e0 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 0x7f4c881a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 0x7f4c881a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49764/0 (socket says 172.21.15.2:49764) 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 -- 172.21.15.2:0/840603452 learned_addr learned my addr 172.21.15.2:0/840603452 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8e400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8810d5c0 0x7f4c8819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8da00700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c881084e0 0x7f4c881a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 -- 172.21.15.2:0/840603452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c881084e0 msgr2=0x7f4c881a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c881084e0 0x7f4c881a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 -- 172.21.15.2:0/840603452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8810d5c0 msgr2=0x7f4c8819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c8810d5c0 0x7f4c8819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740095f0 con 0x7f4c88108e90 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.822+0000 7f4c8d000700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 0x7f4c881a4ab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.823+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7401d070 con 0x7f4c88108e90 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.823+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c7400b610 con 0x7f4c88108e90 2024-03-11T18:31:42.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.823+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8819f3a0 con 0x7f4c88108e90 2024-03-11T18:31:42.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.823+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740173f0 con 0x7f4c88108e90 2024-03-11T18:31:42.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.823+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c881ab4f0 con 0x7f4c88108e90 2024-03-11T18:31:42.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.824+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c881099c0 con 0x7f4c88108e90 2024-03-11T18:31:42.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.826+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f4c74022600 con 0x7f4c88108e90 2024-03-11T18:31:42.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c86000700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c700a1bd0 0x7f4c700a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:42.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4c740cf540 con 0x7f4c88108e90 2024-03-11T18:31:42.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c8da00700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c700a1bd0 0x7f4c700a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:42.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c86000700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] conn(0x7f4c700accb0 0x7f4c700af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:42.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c700af820 con 0x7f4c700accb0 2024-03-11T18:31:42.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c740cf8b0 con 0x7f4c88108e90 2024-03-11T18:31:42.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.828+0000 7f4c8e400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] conn(0x7f4c700accb0 0x7f4c700af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:42.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.829+0000 7f4c8da00700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c700a1bd0 0x7f4c700a4090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009c60 tx=0x7f4c7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:42.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.829+0000 7f4c8e400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] conn(0x7f4c700accb0 0x7f4c700af150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:42.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.830+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== osd.9 v2:172.21.15.2:6826/1412624036 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4c700af820 con 0x7f4c700accb0 2024-03-11T18:31:42.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.849+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 --> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c88118650 con 0x7f4c700accb0 2024-03-11T18:31:42.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.850+0000 7f4c86000700 1 -- 172.21.15.2:0/840603452 <== osd.9 v2:172.21.15.2:6826/1412624036 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4c88118650 con 0x7f4c700accb0 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.850+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] conn(0x7f4c700accb0 msgr2=0x7f4c700af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.850+0000 7f4c90400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] conn(0x7f4c700accb0 0x7f4c700af150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c700a1bd0 msgr2=0x7f4c700a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4c700a1bd0 0x7f4c700a4090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009c60 tx=0x7f4c7c009260 comp rx=0 tx=0).stop 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 msgr2=0x7f4c881a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 --2- 172.21.15.2:0/840603452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c88108e90 0x7f4c881a4ab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74005740 comp rx=0 tx=0).stop 2024-03-11T18:31:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c8e400700 1 -- 172.21.15.2:0/840603452 reap_dead start 2024-03-11T18:31:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 shutdown_connections 2024-03-11T18:31:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.851+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 >> 172.21.15.2:0/840603452 conn(0x7f4c88075440 msgr2=0x7f4c88077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.852+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 shutdown_connections 2024-03-11T18:31:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:42.852+0000 7f4c90400700 1 -- 172.21.15.2:0/840603452 wait complete. 2024-03-11T18:31:43.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 -- 172.21.15.2:0/2552691083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784100b70 msgr2=0x7f2784100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 --2- 172.21.15.2:0/2552691083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784100b70 0x7f2784100ff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2770009990 tx=0x7f2770009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 -- 172.21.15.2:0/2552691083 shutdown_connections 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 --2- 172.21.15.2:0/2552691083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784101530 0x7f278410d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 --2- 172.21.15.2:0/2552691083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2784106b80 0x7f2784106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 --2- 172.21.15.2:0/2552691083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784100b70 0x7f2784100ff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 -- 172.21.15.2:0/2552691083 >> 172.21.15.2:0/2552691083 conn(0x7f2784077aa0 msgr2=0x7f2784077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 -- 172.21.15.2:0/2552691083 shutdown_connections 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.929+0000 7f278b800700 1 -- 172.21.15.2:0/2552691083 wait complete. 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.930+0000 7f278b800700 1 Processor -- start 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.930+0000 7f278b800700 1 -- start start 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784100b70 0x7f27841027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:43.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2784101530 0x7f2784102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 0x7f2784103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a6540 con 0x7f2784106b80 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a66b0 con 0x7f2784101530 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f278b800700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a6870 con 0x7f2784100b70 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2789800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 0x7f2784103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2788e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784100b70 0x7f27841027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2788e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784100b70 0x7f27841027b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48378/0 (socket says 172.21.15.2:48378) 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2789800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 0x7f2784103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49752/0 (socket says 172.21.15.2:49752) 2024-03-11T18:31:43.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2788e00700 1 -- 172.21.15.2:0/1994831903 learned_addr learned my addr 172.21.15.2:0/1994831903 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.931+0000 7f2783e00700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2784101530 0x7f2784102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 -- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784100b70 msgr2=0x7f27841027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2784100b70 0x7f27841027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 -- 172.21.15.2:0/1994831903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2784101530 msgr2=0x7f2784102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2784101530 0x7f2784102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27700095f0 con 0x7f2784106b80 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2789800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 0x7f2784103230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f277c007f10 tx=0x7f277c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:43.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c009ea0 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277c00f040 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c004400 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27841a6b50 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.932+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27841a7000 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.933+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2784110ea0 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.936+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f277c003a00 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.937+0000 7f2781600700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f276c0a1bd0 0x7f276c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.937+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f277c0cde80 con 0x7f2784106b80 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.937+0000 7f2788e00700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f276c0a1bd0 0x7f276c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.937+0000 7f2781600700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] conn(0x7f276c0accb0 0x7f276c0af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.938+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f276c0af820 con 0x7f276c0accb0 2024-03-11T18:31:43.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.938+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f277c0ce1f0 con 0x7f2784106b80 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.938+0000 7f2783e00700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] conn(0x7f276c0accb0 0x7f276c0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.938+0000 7f2788e00700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f276c0a1bd0 0x7f276c0a4090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2778009ca0 tx=0x7f2778009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.939+0000 7f2783e00700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] conn(0x7f276c0accb0 0x7f276c0af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.940+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== osd.2 v2:172.21.15.165:6800/758382799 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f276c0af820 con 0x7f276c0accb0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.958+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 --> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2784104ba0 con 0x7f276c0accb0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f2781600700 1 -- 172.21.15.2:0/1994831903 <== osd.2 v2:172.21.15.165:6800/758382799 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2784104ba0 con 0x7f276c0accb0 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] conn(0x7f276c0accb0 msgr2=0x7f276c0af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] conn(0x7f276c0accb0 0x7f276c0af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f276c0a1bd0 msgr2=0x7f276c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f276c0a1bd0 0x7f276c0a4090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2778009ca0 tx=0x7f2778009260 comp rx=0 tx=0).stop 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 msgr2=0x7f2784103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:43.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f278b800700 1 --2- 172.21.15.2:0/1994831903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2784106b80 0x7f2784103230 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f277c007f10 tx=0x7f277c00d3b0 comp rx=0 tx=0).stop 2024-03-11T18:31:43.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.959+0000 7f2789800700 1 -- 172.21.15.2:0/1994831903 reap_dead start 2024-03-11T18:31:43.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.960+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 shutdown_connections 2024-03-11T18:31:43.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.960+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 >> 172.21.15.2:0/1994831903 conn(0x7f2784077aa0 msgr2=0x7f27840ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:43.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.960+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 shutdown_connections 2024-03-11T18:31:43.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:40.960+0000 7f278b800700 1 -- 172.21.15.2:0/1994831903 wait complete. 2024-03-11T18:31:43.662 INFO:teuthology.orchestra.run.smithi002.stdout:206158430263 2024-03-11T18:31:43.663 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T18:31:43.979 INFO:teuthology.orchestra.run.smithi002.stdout:261993005100 2024-03-11T18:31:43.980 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-11T18:31:44.173 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:44 smithi002 ceph-mon[31328]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s; 94 B/s, 0 objects/s recovering 2024-03-11T18:31:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:44 smithi023 ceph-mon[36406]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s; 94 B/s, 0 objects/s recovering 2024-03-11T18:31:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:44 smithi165 ceph-mon[36634]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s; 94 B/s, 0 objects/s recovering 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 -- 172.21.15.2:0/2222583446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 msgr2=0x7f0f6c10df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 --2- 172.21.15.2:0/2222583446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c10df50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009990 tx=0x7f0f58009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 -- 172.21.15.2:0/2222583446 shutdown_connections 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 --2- 172.21.15.2:0/2222583446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f6c10e490 0x7f0f6c110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 --2- 172.21.15.2:0/2222583446 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f6c0740a0 0x7f0f6c074480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 --2- 172.21.15.2:0/2222583446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c10df50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 -- 172.21.15.2:0/2222583446 >> 172.21.15.2:0/2222583446 conn(0x7f0f6c0fd720 msgr2=0x7f0f6c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:44.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 -- 172.21.15.2:0/2222583446 shutdown_connections 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.668+0000 7f0f72600700 1 -- 172.21.15.2:0/2222583446 wait complete. 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.669+0000 7f0f72600700 1 Processor -- start 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.669+0000 7f0f72600700 1 -- start start 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f6c0740a0 0x7f0f6c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f6c10e490 0x7f0f6c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c19f260 con 0x7f0f6c10e490 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c19f3d0 con 0x7f0f6c0749c0 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f72600700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c19f590 con 0x7f0f6c0740a0 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f6aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f6be00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f6c10e490 0x7f0f6c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f6aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c19a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:57494/0 (socket says 172.21.15.2:57494) 2024-03-11T18:31:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f6aa00700 1 -- 172.21.15.2:0/615351358 learned_addr learned my addr 172.21.15.2:0/615351358 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.670+0000 7f0f6b400700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f6c0740a0 0x7f0f6c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 -- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f6c0740a0 msgr2=0x7f0f6c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f6c0740a0 0x7f0f6c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 -- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f6c10e490 msgr2=0x7f0f6c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0f6c10e490 0x7f0f6c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c009520 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f6aa00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c19a8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f58000c00 tx=0x7f0f58004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5801d070 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f5800b610 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f580095f0 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.671+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f580173f0 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.672+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c1a6f00 con 0x7f0f6c0749c0 2024-03-11T18:31:44.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.672+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f6c1141c0 con 0x7f0f6c0749c0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.675+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f0f58022670 con 0x7f0f6c0749c0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f63e00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0f500a1bd0 0x7f0f500a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0f580cefc0 con 0x7f0f6c0749c0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f6b400700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0f500a1bd0 0x7f0f500a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f63e00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] conn(0x7f0f500accb0 0x7f0f500af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f500af820 con 0x7f0f500accb0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.676+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f580cf330 con 0x7f0f6c0749c0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.677+0000 7f0f6be00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] conn(0x7f0f500accb0 0x7f0f500af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.677+0000 7f0f6b400700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0f500a1bd0 0x7f0f500a4090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009df0 tx=0x7f0f5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:44.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.678+0000 7f0f6be00700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] conn(0x7f0f500accb0 0x7f0f500af150 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.679+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== osd.11 v2:172.21.15.165:6824/1832747584 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0f500af820 con 0x7f0f500accb0 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.698+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 --> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f6c062330 con 0x7f0f500accb0 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f63e00700 1 -- 172.21.15.2:0/615351358 <== osd.11 v2:172.21.15.165:6824/1832747584 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0f6c062330 con 0x7f0f500accb0 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] conn(0x7f0f500accb0 msgr2=0x7f0f500af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] conn(0x7f0f500accb0 0x7f0f500af150 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:44.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0f500a1bd0 msgr2=0x7f0f500a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0f500a1bd0 0x7f0f500a4090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009df0 tx=0x7f0f5c009260 comp rx=0 tx=0).stop 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 msgr2=0x7f0f6c19a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 --2- 172.21.15.2:0/615351358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f6c0749c0 0x7f0f6c19a8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f58000c00 tx=0x7f0f58004730 comp rx=0 tx=0).stop 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f6be00700 1 -- 172.21.15.2:0/615351358 reap_dead start 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 shutdown_connections 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.699+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 >> 172.21.15.2:0/615351358 conn(0x7f0f6c0fd720 msgr2=0x7f0f6c0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.700+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 shutdown_connections 2024-03-11T18:31:44.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:44.700+0000 7f0f72600700 1 -- 172.21.15.2:0/615351358 wait complete. 2024-03-11T18:31:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 -- 172.21.15.2:0/2495253466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc101f20 msgr2=0x7f45bc102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 --2- 172.21.15.2:0/2495253466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc101f20 0x7f45bc102300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009990 tx=0x7f45b0009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 -- 172.21.15.2:0/2495253466 shutdown_connections 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 --2- 172.21.15.2:0/2495253466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc106ff0 0x7f45bc10ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 --2- 172.21.15.2:0/2495253466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc101f20 0x7f45bc102300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 --2- 172.21.15.2:0/2495253466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 0x7f45bc106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 -- 172.21.15.2:0/2495253466 >> 172.21.15.2:0/2495253466 conn(0x7f45bc078dd0 msgr2=0x7f45bc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 -- 172.21.15.2:0/2495253466 shutdown_connections 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.364+0000 7f45c3400700 1 -- 172.21.15.2:0/2495253466 wait complete. 2024-03-11T18:31:45.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.366+0000 7f45c3400700 1 Processor -- start 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.366+0000 7f45c3400700 1 -- start start 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 0x7f45bc19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 0x7f45bc19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc106ff0 0x7f45bc19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc19f3a0 con 0x7f45bc101f20 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc19f510 con 0x7f45bc106ff0 2024-03-11T18:31:45.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c3400700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc19f6d0 con 0x7f45bc1028d0 2024-03-11T18:31:45.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c0a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 0x7f45bc19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:45.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45bbe00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 0x7f45bc19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c0a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 0x7f45bc19a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49798/0 (socket says 172.21.15.2:49798) 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c0a00700 1 -- 172.21.15.2:0/1720399748 learned_addr learned my addr 172.21.15.2:0/1720399748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.367+0000 7f45c1400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc106ff0 0x7f45bc19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 -- 172.21.15.2:0/1720399748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 msgr2=0x7f45bc19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 0x7f45bc19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 -- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc106ff0 msgr2=0x7f45bc19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45bc106ff0 0x7f45bc19ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8009520 con 0x7f45bc101f20 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45bbe00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc1028d0 0x7f45bc19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:45.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c0a00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 0x7f45bc19a510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f45b0000c00 tx=0x7f45b0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b001d070 con 0x7f45bc101f20 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b00095f0 con 0x7f45bc101f20 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45b000b610 con 0x7f45bc101f20 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.368+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc1a7040 con 0x7f45bc101f20 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.369+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b000fb00 con 0x7f45bc101f20 2024-03-11T18:31:45.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.369+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45bc112020 con 0x7f45bc101f20 2024-03-11T18:31:45.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.373+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f45b000fc60 con 0x7f45bc101f20 2024-03-11T18:31:45.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45b9600700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f45a40a1b50 0x7f45a40a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:45.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f45b00cf270 con 0x7f45bc101f20 2024-03-11T18:31:45.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45bbe00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f45a40a1b50 0x7f45a40a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:45.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45b9600700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] conn(0x7f45a40acc30 0x7f45a40af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:45.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45a40af7a0 con 0x7f45a40acc30 2024-03-11T18:31:45.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.375+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45b00cf5e0 con 0x7f45bc101f20 2024-03-11T18:31:45.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.376+0000 7f45c1400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] conn(0x7f45a40acc30 0x7f45a40af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:45.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.376+0000 7f45bbe00700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f45a40a1b50 0x7f45a40a4010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009fd0 tx=0x7f45a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:45.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.376+0000 7f45c1400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] conn(0x7f45a40acc30 0x7f45a40af0d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:45.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.377+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== osd.10 v2:172.21.15.23:6824/1131591289 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f45a40af7a0 con 0x7f45a40acc30 2024-03-11T18:31:45.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.395+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 --> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f45bc100630 con 0x7f45a40acc30 2024-03-11T18:31:45.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45b9600700 1 -- 172.21.15.2:0/1720399748 <== osd.10 v2:172.21.15.23:6824/1131591289 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f45bc100630 con 0x7f45a40acc30 2024-03-11T18:31:45.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] conn(0x7f45a40acc30 msgr2=0x7f45a40af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] conn(0x7f45a40acc30 0x7f45a40af0d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:45.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f45a40a1b50 msgr2=0x7f45a40a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f45a40a1b50 0x7f45a40a4010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009fd0 tx=0x7f45a8009260 comp rx=0 tx=0).stop 2024-03-11T18:31:45.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 msgr2=0x7f45bc19a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:45.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 --2- 172.21.15.2:0/1720399748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45bc101f20 0x7f45bc19a510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f45b0000c00 tx=0x7f45b0005740 comp rx=0 tx=0).stop 2024-03-11T18:31:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c1400700 1 -- 172.21.15.2:0/1720399748 reap_dead start 2024-03-11T18:31:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 shutdown_connections 2024-03-11T18:31:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.396+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 >> 172.21.15.2:0/1720399748 conn(0x7f45bc078dd0 msgr2=0x7f45bc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.397+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 shutdown_connections 2024-03-11T18:31:45.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:45.397+0000 7f45c3400700 1 -- 172.21.15.2:0/1720399748 wait complete. 2024-03-11T18:31:46.079 INFO:teuthology.orchestra.run.smithi002.stdout:77309411409 2024-03-11T18:31:46.079 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T18:31:46.362 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:46 smithi002 ceph-mon[31328]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:31:46.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:46 smithi165 ceph-mon[36634]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:31:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:46 smithi023 ceph-mon[36406]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:31:46.406 INFO:teuthology.orchestra.run.smithi002.stdout:171798691903 2024-03-11T18:31:46.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T18:31:46.785 INFO:teuthology.orchestra.run.smithi002.stdout:296352743464 2024-03-11T18:31:46.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-11T18:31:47.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.081+0000 7f8a8ee00700 1 -- 172.21.15.2:0/3769381340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 msgr2=0x7f8a88106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.081+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/3769381340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88106910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009990 tx=0x7f8a74009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 -- 172.21.15.2:0/3769381340 shutdown_connections 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/3769381340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a88106ff0 0x7f8a8810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/3769381340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 0x7f8a88102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/3769381340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88106910 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 -- 172.21.15.2:0/3769381340 >> 172.21.15.2:0/3769381340 conn(0x7f8a88078dd0 msgr2=0x7f8a880791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 -- 172.21.15.2:0/3769381340 shutdown_connections 2024-03-11T18:31:47.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.082+0000 7f8a8ee00700 1 -- 172.21.15.2:0/3769381340 wait complete. 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.083+0000 7f8a8ee00700 1 Processor -- start 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.083+0000 7f8a8ee00700 1 -- start start 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.083+0000 7f8a8ee00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 0x7f8a881982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.083+0000 7f8a8ee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a8ee00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a88106ff0 0x7f8a8819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a8ee00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819d1a0 con 0x7f8a88101f20 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a8ee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819d310 con 0x7f8a881028d0 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a8ee00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819d4d0 con 0x7f8a88106ff0 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 0x7f8a881982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:47.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 0x7f8a881982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49820/0 (socket says 172.21.15.2:49820) 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87e00700 1 -- 172.21.15.2:0/1282925706 learned_addr learned my addr 172.21.15.2:0/1282925706 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a8ce00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a88106ff0 0x7f8a8819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 -- 172.21.15.2:0/1282925706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a88106ff0 msgr2=0x7f8a8819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a88106ff0 0x7f8a8819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 -- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 msgr2=0x7f8a881982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a88101f20 0x7f8a881982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.084+0000 7f8a87400700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a740095f0 con 0x7f8a881028d0 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a87400700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88198800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a8806a7d0 tx=0x7f8a74005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7401d070 con 0x7f8a881028d0 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a7400b610 con 0x7f8a881028d0 2024-03-11T18:31:47.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a740173f0 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8819d750 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.085+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a88074070 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.086+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a88112040 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.088+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f8a74022670 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.089+0000 7f8a84c00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f8a6c0a1b50 0x7f8a6c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.089+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f8a740cf420 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a87e00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f8a6c0a1b50 0x7f8a6c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a84c00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] conn(0x7f8a6c0acc30 0x7f8a6c0af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a6c0af7a0 con 0x7f8a6c0acc30 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a740cf790 con 0x7f8a881028d0 2024-03-11T18:31:47.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a8ce00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] conn(0x7f8a6c0acc30 0x7f8a6c0af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:47.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.090+0000 7f8a87e00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f8a6c0a1b50 0x7f8a6c0a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8a78005950 tx=0x7f8a7800f910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:47.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.091+0000 7f8a8ce00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] conn(0x7f8a6c0acc30 0x7f8a6c0af0d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:47.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.092+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== osd.0 v2:172.21.15.2:6802/1098127519 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8a6c0af7a0 con 0x7f8a6c0acc30 2024-03-11T18:31:47.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.110+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 --> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a88062380 con 0x7f8a6c0acc30 2024-03-11T18:31:47.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a84c00700 1 -- 172.21.15.2:0/1282925706 <== osd.0 v2:172.21.15.2:6802/1098127519 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a88062380 con 0x7f8a6c0acc30 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] conn(0x7f8a6c0acc30 msgr2=0x7f8a6c0af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] conn(0x7f8a6c0acc30 0x7f8a6c0af0d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f8a6c0a1b50 msgr2=0x7f8a6c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f8a6c0a1b50 0x7f8a6c0a4010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8a78005950 tx=0x7f8a7800f910 comp rx=0 tx=0).stop 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 msgr2=0x7f8a88198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.111+0000 7f8a8ee00700 1 --2- 172.21.15.2:0/1282925706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a881028d0 0x7f8a88198800 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a8806a7d0 tx=0x7f8a74005740 comp rx=0 tx=0).stop 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.112+0000 7f8a8ce00700 1 -- 172.21.15.2:0/1282925706 reap_dead start 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.112+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 shutdown_connections 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.112+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 >> 172.21.15.2:0/1282925706 conn(0x7f8a88078dd0 msgr2=0x7f8a88101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.112+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 shutdown_connections 2024-03-11T18:31:47.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:46.112+0000 7f8a8ee00700 1 -- 172.21.15.2:0/1282925706 wait complete. 2024-03-11T18:31:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:48 smithi165 ceph-mon[36634]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:31:48.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:48 smithi023 ceph-mon[36406]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:31:48.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:48 smithi002 ceph-mon[31328]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:31:48.763 INFO:teuthology.orchestra.run.smithi002.stdout:34359738460 2024-03-11T18:31:48.763 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T18:31:48.961 INFO:teuthology.orchestra.run.smithi002.stdout:55834574932 2024-03-11T18:31:48.961 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T18:31:49.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 -- 172.21.15.2:0/2440755298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e580747c0 msgr2=0x7f4e5810e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 --2- 172.21.15.2:0/2440755298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e580747c0 0x7f4e5810e130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009990 tx=0x7f4e44009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 -- 172.21.15.2:0/2440755298 shutdown_connections 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 --2- 172.21.15.2:0/2440755298 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e58073ea0 0x7f4e58074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 --2- 172.21.15.2:0/2440755298 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e5810e670 0x7f4e58110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 --2- 172.21.15.2:0/2440755298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e580747c0 0x7f4e5810e130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 -- 172.21.15.2:0/2440755298 >> 172.21.15.2:0/2440755298 conn(0x7f4e58078dd0 msgr2=0x7f4e580791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.968+0000 7f4e5d600700 1 -- 172.21.15.2:0/2440755298 shutdown_connections 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.969+0000 7f4e5d600700 1 -- 172.21.15.2:0/2440755298 wait complete. 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 Processor -- start 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 -- start start 2024-03-11T18:31:49.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 0x7f4e58101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e580747c0 0x7f4e581022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e5810e670 0x7f4e58102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581aaaa0 con 0x7f4e5810e670 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581aabe0 con 0x7f4e58073ea0 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.970+0000 7f4e5d600700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581aad20 con 0x7f4e580747c0 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 0x7f4e58101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 0x7f4e58101d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47606/0 (socket says 172.21.15.2:47606) 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 -- 172.21.15.2:0/1645079488 learned_addr learned my addr 172.21.15.2:0/1645079488 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56000700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e580747c0 0x7f4e581022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:49.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e57400700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e5810e670 0x7f4e58102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 -- 172.21.15.2:0/1645079488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e580747c0 msgr2=0x7f4e581022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e580747c0 0x7f4e581022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 -- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e5810e670 msgr2=0x7f4e58102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e5810e670 0x7f4e58102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.971+0000 7f4e56a00700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440095f0 con 0x7f4e58073ea0 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e56a00700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 0x7f4e58101d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800e9d0 tx=0x7f4e4800ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48009790 con 0x7f4e58073ea0 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e4800ee70 con 0x7f4e58073ea0 2024-03-11T18:31:49.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48018680 con 0x7f4e58073ea0 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e581aafa0 con 0x7f4e58073ea0 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.972+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e581ab420 con 0x7f4e58073ea0 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.973+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4e58114270 con 0x7f4e58073ea0 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.976+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f4e4800cbd0 con 0x7f4e58073ea0 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.978+0000 7f4e4f400700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4e3c0a1bd0 0x7f4e3c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:49.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.979+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4e48014070 con 0x7f4e58073ea0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.979+0000 7f4e56000700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4e3c0a1bd0 0x7f4e3c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.979+0000 7f4e4f400700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] conn(0x7f4e3c0accb0 0x7f4e3c0af150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.979+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e3c0af820 con 0x7f4e3c0accb0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.979+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e480ce0a0 con 0x7f4e58073ea0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.980+0000 7f4e57400700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] conn(0x7f4e3c0accb0 0x7f4e3c0af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.980+0000 7f4e56000700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4e3c0a1bd0 0x7f4e3c0a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4e440098e0 tx=0x7f4e44005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.980+0000 7f4e57400700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] conn(0x7f4e3c0accb0 0x7f4e3c0af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:48.981+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== osd.4 v2:172.21.15.23:6808/2973022110 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4e3c0af820 con 0x7f4e3c0accb0 2024-03-11T18:31:49.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.004+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 --> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4e58100830 con 0x7f4e3c0accb0 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e4f400700 1 -- 172.21.15.2:0/1645079488 <== osd.4 v2:172.21.15.23:6808/2973022110 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4e58100830 con 0x7f4e3c0accb0 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] conn(0x7f4e3c0accb0 msgr2=0x7f4e3c0af150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] conn(0x7f4e3c0accb0 0x7f4e3c0af150 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4e3c0a1bd0 msgr2=0x7f4e3c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f4e3c0a1bd0 0x7f4e3c0a4090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4e440098e0 tx=0x7f4e44005960 comp rx=0 tx=0).stop 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 msgr2=0x7f4e58101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e5d600700 1 --2- 172.21.15.2:0/1645079488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e58073ea0 0x7f4e58101d90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800e9d0 tx=0x7f4e4800ece0 comp rx=0 tx=0).stop 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.005+0000 7f4e57400700 1 -- 172.21.15.2:0/1645079488 reap_dead start 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.006+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 shutdown_connections 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.006+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 >> 172.21.15.2:0/1645079488 conn(0x7f4e58078dd0 msgr2=0x7f4e580ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:49.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.006+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 shutdown_connections 2024-03-11T18:31:49.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:49.006+0000 7f4e5d600700 1 -- 172.21.15.2:0/1645079488 wait complete. 2024-03-11T18:31:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:50 smithi002 ceph-mon[31328]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-11T18:31:50.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:50 smithi165 ceph-mon[36634]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-11T18:31:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:50 smithi023 ceph-mon[36406]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-11T18:31:50.533 INFO:teuthology.orchestra.run.smithi002.stdout:317827579938 2024-03-11T18:31:50.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-11T18:31:50.981 INFO:teuthology.orchestra.run.smithi002.stdout:352187318300 2024-03-11T18:31:50.982 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 -- 172.21.15.2:0/2478932289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6948101f20 msgr2=0x7f6948102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 --2- 172.21.15.2:0/2478932289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6948101f20 0x7f6948102300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6938009990 tx=0x7f6938009ca0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 -- 172.21.15.2:0/2478932289 shutdown_connections 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 --2- 172.21.15.2:0/2478932289 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 0x7f694810ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 --2- 172.21.15.2:0/2478932289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6948101f20 0x7f6948102300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 --2- 172.21.15.2:0/2478932289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69481028d0 0x7f6948106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 -- 172.21.15.2:0/2478932289 >> 172.21.15.2:0/2478932289 conn(0x7f6948078dd0 msgr2=0x7f69480791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 -- 172.21.15.2:0/2478932289 shutdown_connections 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.210+0000 7f694f000700 1 -- 172.21.15.2:0/2478932289 wait complete. 2024-03-11T18:31:51.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.211+0000 7f694f000700 1 Processor -- start 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 -- start start 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6948101f20 0x7f69481982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69481028d0 0x7f69481987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 0x7f694819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694819d130 con 0x7f69481028d0 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694819d2a0 con 0x7f6948101f20 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694f000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694819d460 con 0x7f6948106ff0 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f6947400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69481028d0 0x7f69481987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f694d000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 0x7f694819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f6947400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69481028d0 0x7f69481987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42606/0 (socket says 172.21.15.2:42606) 2024-03-11T18:31:51.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f6947400700 1 -- 172.21.15.2:0/850155042 learned_addr learned my addr 172.21.15.2:0/850155042 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.212+0000 7f6947e00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6948101f20 0x7f69481982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 -- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6948101f20 msgr2=0x7f69481982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6948101f20 0x7f69481982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 -- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69481028d0 msgr2=0x7f69481987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69481028d0 0x7f69481987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69380095f0 con 0x7f6948106ff0 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f6947e00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6948101f20 0x7f69481982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:51.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.213+0000 7f694d000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 0x7f694819cb20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6940007f10 tx=0x7f694000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.214+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940004db0 con 0x7f6948106ff0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.214+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694819d740 con 0x7f6948106ff0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.214+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f694000f040 con 0x7f6948106ff0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.214+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400059d0 con 0x7f6948106ff0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.214+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481a4fa0 con 0x7f6948106ff0 2024-03-11T18:31:51.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.215+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6948112020 con 0x7f6948106ff0 2024-03-11T18:31:51.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.217+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f694000db20 con 0x7f6948106ff0 2024-03-11T18:31:51.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.218+0000 7f6944c00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f692c0a1b50 0x7f692c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:51.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.218+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f69400ccff0 con 0x7f6948106ff0 2024-03-11T18:31:51.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.219+0000 7f6947e00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f692c0a1b50 0x7f692c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.219+0000 7f6944c00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] conn(0x7f692c0acc30 0x7f692c0af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.219+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f692c0af7a0 con 0x7f692c0acc30 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.219+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69400cf5d0 con 0x7f6948106ff0 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.219+0000 7f6947400700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] conn(0x7f692c0acc30 0x7f692c0af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.220+0000 7f6947e00700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f692c0a1b50 0x7f692c0a4010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f693800b2e0 tx=0x7f6938005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:51.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.220+0000 7f6947400700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] conn(0x7f692c0acc30 0x7f692c0af0d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:51.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.221+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== osd.7 v2:172.21.15.23:6816/2299093743 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f692c0af7a0 con 0x7f692c0acc30 2024-03-11T18:31:51.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.240+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 --> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f69480623b0 con 0x7f692c0acc30 2024-03-11T18:31:51.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.241+0000 7f6944c00700 1 -- 172.21.15.2:0/850155042 <== osd.7 v2:172.21.15.23:6816/2299093743 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f69480623b0 con 0x7f692c0acc30 2024-03-11T18:31:51.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.241+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] conn(0x7f692c0acc30 msgr2=0x7f692c0af0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.241+0000 7f694f000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] conn(0x7f692c0acc30 0x7f692c0af0d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f692c0a1b50 msgr2=0x7f692c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f692c0a1b50 0x7f692c0a4010 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f693800b2e0 tx=0x7f6938005fd0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 msgr2=0x7f694819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 --2- 172.21.15.2:0/850155042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6948106ff0 0x7f694819cb20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6940007f10 tx=0x7f694000d3b0 comp rx=0 tx=0).stop 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694d000700 1 -- 172.21.15.2:0/850155042 reap_dead start 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 shutdown_connections 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 >> 172.21.15.2:0/850155042 conn(0x7f6948078dd0 msgr2=0x7f69480fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 shutdown_connections 2024-03-11T18:31:51.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:51.242+0000 7f694f000700 1 -- 172.21.15.2:0/850155042 wait complete. 2024-03-11T18:31:51.722 INFO:teuthology.orchestra.run.smithi002.stdout:150323855430 2024-03-11T18:31:51.722 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T18:31:52.299 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:52.304 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:52.310 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:52.330 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:52.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:52 smithi002 ceph-mon[31328]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:31:52.379 INFO:teuthology.orchestra.run.smithi002.stdout:227633266741 2024-03-11T18:31:52.379 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T18:31:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:52 smithi165 ceph-mon[36634]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:31:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:52 smithi023 ceph-mon[36406]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:31:53.355 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:54.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:54 smithi002 ceph-mon[31328]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:31:54.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:54 smithi165 ceph-mon[36634]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:31:54.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:54 smithi023 ceph-mon[36406]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:31:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:56 smithi165 ceph-mon[36634]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:31:56.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:56 smithi023 ceph-mon[36406]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:31:56.422 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:56 smithi002 ceph-mon[31328]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:31:57.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 -- 172.21.15.2:0/4143063811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81084e0 msgr2=0x7f99c81088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:57.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 --2- 172.21.15.2:0/4143063811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81084e0 0x7f99c81088c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f99b80098f0 tx=0x7f99b8009c00 comp rx=0 tx=0).stop 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 -- 172.21.15.2:0/4143063811 shutdown_connections 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 --2- 172.21.15.2:0/4143063811 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c810d5c0 0x7f99c8115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 --2- 172.21.15.2:0/4143063811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c81084e0 0x7f99c81088c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 --2- 172.21.15.2:0/4143063811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 0x7f99c810cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 -- 172.21.15.2:0/4143063811 >> 172.21.15.2:0/4143063811 conn(0x7f99c8075440 msgr2=0x7f99c8077860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 -- 172.21.15.2:0/4143063811 shutdown_connections 2024-03-11T18:31:57.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.289+0000 7f99cee00700 1 -- 172.21.15.2:0/4143063811 wait complete. 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.290+0000 7f99cee00700 1 Processor -- start 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 -- start start 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 0x7f99c81a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c819ee10 con 0x7f99c81084e0 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c819ef80 con 0x7f99c810d5c0 2024-03-11T18:31:57.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99cee00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c819f140 con 0x7f99c8108e90 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.291+0000 7f99c7e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99c7400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 0x7f99c81a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56064/0 (socket says 172.21.15.2:56064) 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99c7e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47798/0 (socket says 172.21.15.2:47798) 2024-03-11T18:31:57.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 -- 172.21.15.2:0/1113301230 learned_addr learned my addr 172.21.15.2:0/1113301230 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 -- 172.21.15.2:0/1113301230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 msgr2=0x7f99c81a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 0x7f99c81a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 -- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 msgr2=0x7f99c81a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 -- 172.21.15.2:0/1113301230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8009520 con 0x7f99c810d5c0 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99c7e00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.292+0000 7f99cce00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99c000c110 tx=0x7f99c000c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.293+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0017070 con 0x7f99c810d5c0 2024-03-11T18:31:57.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.293+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99c000e050 con 0x7f99c810d5c0 2024-03-11T18:31:57.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.293+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99c0007940 con 0x7f99c810d5c0 2024-03-11T18:31:57.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.293+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c819f3c0 con 0x7f99c810d5c0 2024-03-11T18:31:57.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.293+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c81ab510 con 0x7f99c810d5c0 2024-03-11T18:31:57.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.294+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c8110880 con 0x7f99c810d5c0 2024-03-11T18:31:57.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.296+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f99c0004ad0 con 0x7f99c810d5c0 2024-03-11T18:31:57.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.297+0000 7f99c4c00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 0x7f99ac0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:31:57.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.297+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f99c00cd2a0 con 0x7f99c810d5c0 2024-03-11T18:31:57.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.297+0000 7f99c7e00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 0x7f99ac0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:31:57.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.298+0000 7f99c7e00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 0x7f99ac0a4090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009810 tx=0x7f99b8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:31:57.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.302+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99c0095fb0 con 0x7f99c810d5c0 2024-03-11T18:31:57.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.538+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f99c8068930 con 0x7f99c810d5c0 2024-03-11T18:31:57.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.539+0000 7f99c4c00700 1 -- 172.21.15.2:0/1113301230 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f99c0099670 con 0x7f99c810d5c0 2024-03-11T18:31:57.541 INFO:teuthology.orchestra.run.smithi002.stdout:115964117069 2024-03-11T18:31:57.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 msgr2=0x7f99ac0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:57.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 0x7f99ac0a4090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f99b8009810 tx=0x7f99b8019040 comp rx=0 tx=0).stop 2024-03-11T18:31:57.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 msgr2=0x7f99c819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:31:57.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99c000c110 tx=0x7f99c000c420 comp rx=0 tx=0).stop 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 shutdown_connections 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99c810d5c0 0x7f99c819e660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99c81084e0 0x7f99c81a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f99ac0a1bd0 0x7f99ac0a4090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 --2- 172.21.15.2:0/1113301230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f99c8108e90 0x7f99c81a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 >> 172.21.15.2:0/1113301230 conn(0x7f99c8075440 msgr2=0x7f99c80777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 shutdown_connections 2024-03-11T18:31:57.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:57.542+0000 7f99cee00700 1 -- 172.21.15.2:0/1113301230 wait complete. 2024-03-11T18:31:58.351 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:58 smithi002 ceph-mon[31328]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:31:58.352 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:58 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1113301230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T18:31:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:58 smithi165 ceph-mon[36634]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:31:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:58 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1113301230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T18:31:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:58 smithi023 ceph-mon[36406]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:31:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:58 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1113301230' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T18:31:58.683 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:58.690 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:58.693 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:58.708 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:31:59.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:59 smithi002 ceph-mon[31328]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-11T18:31:59.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:31:59 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2334229439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T18:31:59.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:59 smithi165 ceph-mon[36634]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-11T18:31:59.425 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:31:59 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2334229439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T18:31:59.425 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:59 smithi023 ceph-mon[36406]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-11T18:31:59.425 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:31:59 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2334229439' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T18:32:00.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 -- 172.21.15.2:0/4061381594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2750073f90 msgr2=0x7f2750074370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:00.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 --2- 172.21.15.2:0/4061381594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2750073f90 0x7f2750074370 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f274000b0a0 tx=0x7f274000b3b0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 -- 172.21.15.2:0/4061381594 shutdown_connections 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 --2- 172.21.15.2:0/4061381594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f275010e550 0x7f2750110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 --2- 172.21.15.2:0/4061381594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2750073f90 0x7f2750074370 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 --2- 172.21.15.2:0/4061381594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 0x7f275010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 -- 172.21.15.2:0/4061381594 >> 172.21.15.2:0/4061381594 conn(0x7f27500fd740 msgr2=0x7f27500ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 -- 172.21.15.2:0/4061381594 shutdown_connections 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.554+0000 7f2754a00700 1 -- 172.21.15.2:0/4061381594 wait complete. 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.555+0000 7f2754a00700 1 Processor -- start 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.555+0000 7f2754a00700 1 -- start start 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 0x7f275019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275010e550 0x7f275019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275019f420 con 0x7f2750073f90 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275019f590 con 0x7f275010e550 2024-03-11T18:32:00.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2754a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275019f750 con 0x7f27500748b0 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47826/0 (socket says 172.21.15.2:47826) 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 -- 172.21.15.2:0/1721557059 learned_addr learned my addr 172.21.15.2:0/1721557059 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f2745400700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 0x7f275019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274e800700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275010e550 0x7f275019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 -- 172.21.15.2:0/1721557059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 msgr2=0x7f275019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 0x7f275019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 -- 172.21.15.2:0/1721557059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275010e550 msgr2=0x7f275019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275010e550 0x7f275019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.556+0000 7f274de00700 1 -- 172.21.15.2:0/1721557059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2748009520 con 0x7f2750073f90 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f274de00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f274000b070 tx=0x7f274000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274000e040 con 0x7f2750073f90 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f274001d470 con 0x7f2750073f90 2024-03-11T18:32:00.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2740009d00 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274001b6a0 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.557+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27501a70c0 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.559+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2750109760 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.561+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f274001d5e0 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.563+0000 7f2747400700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 0x7f27340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.563+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f27400ce730 con 0x7f2750073f90 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.563+0000 7f2745400700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 0x7f27340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.564+0000 7f2745400700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 0x7f27340a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f275010f800 tx=0x7f2738009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:00.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.568+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f27400974d0 con 0x7f2750073f90 2024-03-11T18:32:00.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.798+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f275019b8a0 con 0x7f2750073f90 2024-03-11T18:32:00.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.798+0000 7f2747400700 1 -- 172.21.15.2:0/1721557059 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f274009ab90 con 0x7f2750073f90 2024-03-11T18:32:00.801 INFO:teuthology.orchestra.run.smithi002.stdout:317827579941 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.801+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 msgr2=0x7f27340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.801+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 0x7f27340a4010 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f275010f800 tx=0x7f2738009260 comp rx=0 tx=0).stop 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.801+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 msgr2=0x7f275019a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.801+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f274000b070 tx=0x7f274000be80 comp rx=0 tx=0).stop 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.801+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 shutdown_connections 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275010e550 0x7f275019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2750073f90 0x7f275019a590 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f27340a1b50 0x7f27340a4010 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 --2- 172.21.15.2:0/1721557059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27500748b0 0x7f275019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 >> 172.21.15.2:0/1721557059 conn(0x7f27500fd740 msgr2=0x7f27500fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 shutdown_connections 2024-03-11T18:32:00.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:00.802+0000 7f2754a00700 1 -- 172.21.15.2:0/1721557059 wait complete. 2024-03-11T18:32:00.970 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117064 got 115964117069 for osd.3 2024-03-11T18:32:00.970 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:01.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.861+0000 7f3f57200700 1 -- 172.21.15.2:0/673632395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 msgr2=0x7f3f50110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.861+0000 7f3f57200700 1 --2- 172.21.15.2:0/673632395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f50110800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b0a0 tx=0x7f3f4c00b3b0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 -- 172.21.15.2:0/673632395 shutdown_connections 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 --2- 172.21.15.2:0/673632395 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f50073f90 0x7f3f50074370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 --2- 172.21.15.2:0/673632395 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f50110800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 --2- 172.21.15.2:0/673632395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f500748b0 0x7f3f5010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 -- 172.21.15.2:0/673632395 >> 172.21.15.2:0/673632395 conn(0x7f3f500fd660 msgr2=0x7f3f500ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 -- 172.21.15.2:0/673632395 shutdown_connections 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.862+0000 7f3f57200700 1 -- 172.21.15.2:0/673632395 wait complete. 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 Processor -- start 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 -- start start 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f50073f90 0x7f3f5019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:01.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f500748b0 0x7f3f5019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5019f290 con 0x7f3f5010e410 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5019f400 con 0x7f3f50073f90 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.863+0000 7f3f57200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5019f5c0 con 0x7f3f500748b0 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47816/0 (socket says 172.21.15.2:47816) 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 -- 172.21.15.2:0/2334229439 learned_addr learned my addr 172.21.15.2:0/2334229439 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f55400700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f500748b0 0x7f3f5019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f55e00700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f50073f90 0x7f3f5019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 -- 172.21.15.2:0/2334229439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f500748b0 msgr2=0x7f3f5019a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f500748b0 0x7f3f5019a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 -- 172.21.15.2:0/2334229439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f50073f90 msgr2=0x7f3f5019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:01.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f50073f90 0x7f3f5019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.864+0000 7f3f56800700 1 -- 172.21.15.2:0/2334229439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40009520 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.865+0000 7f3f56800700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b070 tx=0x7f3f4c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.865+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c00e040 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.865+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c01d470 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.865+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c009d00 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.865+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f501a6f30 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.866+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c005260 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.867+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f50109620 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.870+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f3f4c01d950 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.872+0000 7f3f4aa00700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 0x7f3f380a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.872+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3f4c0cee70 con 0x7f3f5010e410 2024-03-11T18:32:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.872+0000 7f3f55e00700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 0x7f3f380a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:01.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.873+0000 7f3f55e00700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 0x7f3f380a3f70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009c60 tx=0x7f3f44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:01.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:58.874+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f4c097c70 con 0x7f3f5010e410 2024-03-11T18:32:01.032 INFO:teuthology.orchestra.run.smithi002.stdout:77309411413 2024-03-11T18:32:01.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.107+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3f50068930 con 0x7f3f5010e410 2024-03-11T18:32:01.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.107+0000 7f3f4aa00700 1 -- 172.21.15.2:0/2334229439 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f4c09b330 con 0x7f3f5010e410 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 msgr2=0x7f3f380a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 0x7f3f380a3f70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009c60 tx=0x7f3f44009260 comp rx=0 tx=0).stop 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 msgr2=0x7f3f5019ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00b070 tx=0x7f3f4c00be20 comp rx=0 tx=0).stop 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 shutdown_connections 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f5010e410 0x7f3f5019ec80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f50073f90 0x7f3f5019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f380a1ab0 0x7f3f380a3f70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 --2- 172.21.15.2:0/2334229439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f500748b0 0x7f3f5019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:01.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.110+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 >> 172.21.15.2:0/2334229439 conn(0x7f3f500fd660 msgr2=0x7f3f500fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:01.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.111+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 shutdown_connections 2024-03-11T18:32:01.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:31:59.111+0000 7f3f57200700 1 -- 172.21.15.2:0/2334229439 wait complete. 2024-03-11T18:32:01.263 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:01.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:01 smithi002 ceph-mon[31328]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-11T18:32:01.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:01 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1721557059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T18:32:01.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:01 smithi165 ceph-mon[36634]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-11T18:32:01.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:01 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1721557059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T18:32:01.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:01 smithi023 ceph-mon[36406]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-11T18:32:01.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:01 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1721557059' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 -- 172.21.15.2:0/3430653852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c107cc0 msgr2=0x7fa08c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 --2- 172.21.15.2:0/3430653852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c107cc0 0x7fa08c1080a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0099c0 tx=0x7fa07c009cd0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 -- 172.21.15.2:0/3430653852 shutdown_connections 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 --2- 172.21.15.2:0/3430653852 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c107cc0 0x7fa08c1080a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 --2- 172.21.15.2:0/3430653852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c06a2c0 0x7fa08c10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 --2- 172.21.15.2:0/3430653852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 -- 172.21.15.2:0/3430653852 >> 172.21.15.2:0/3430653852 conn(0x7fa08c077aa0 msgr2=0x7fa08c077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 -- 172.21.15.2:0/3430653852 shutdown_connections 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.401+0000 7fa090e00700 1 -- 172.21.15.2:0/3430653852 wait complete. 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.403+0000 7fa090e00700 1 Processor -- start 2024-03-11T18:32:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.403+0000 7fa090e00700 1 -- start start 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 0x7fa08c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c19f300 con 0x7fa08c107cc0 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c19f470 con 0x7fa08c069920 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa090e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c19f630 con 0x7fa08c06a2c0 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa08a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa089e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa089400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 0x7fa08c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa089e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56120/0 (socket says 172.21.15.2:56120) 2024-03-11T18:32:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa08a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47854/0 (socket says 172.21.15.2:47854) 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.404+0000 7fa089e00700 1 -- 172.21.15.2:0/1961245004 learned_addr learned my addr 172.21.15.2:0/1961245004 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 -- 172.21.15.2:0/1961245004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 msgr2=0x7fa08c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stdout:296352743468 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 -- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 msgr2=0x7fa08c19ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 -- 172.21.15.2:0/1961245004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c0095f0 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa08a800700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.405+0000 7fa089400700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 0x7fa08c19a9b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa07800e9f0 tx=0x7fa07800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.406+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07800c9c0 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.406+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa078010430 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.406+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c19f910 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.406+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa078018680 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.406+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c1a7170 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.407+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c068930 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.409+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fa078010910 con 0x7fa08c06a2c0 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.411+0000 7fa082a00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 0x7fa0700a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.411+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fa078014070 con 0x7fa08c06a2c0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.411+0000 7fa089e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 0x7fa0700a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.412+0000 7fa089e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 0x7fa0700a4090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa07c000c00 tx=0x7fa07c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.413+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa078096710 con 0x7fa08c06a2c0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.659+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fa08c111ce0 con 0x7fa08c06a2c0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.660+0000 7fa082a00700 1 -- 172.21.15.2:0/1961245004 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa0780105a0 con 0x7fa08c06a2c0 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 msgr2=0x7fa0700a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 0x7fa0700a4090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa07c000c00 tx=0x7fa07c005fd0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 msgr2=0x7fa08c19a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 0x7fa08c19a9b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa07800e9f0 tx=0x7fa07800ed00 comp rx=0 tx=0).stop 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 shutdown_connections 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa08c107cc0 0x7fa08c19ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa08c069920 0x7fa08c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa0700a1bd0 0x7fa0700a4090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 --2- 172.21.15.2:0/1961245004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa08c06a2c0 0x7fa08c19a9b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 >> 172.21.15.2:0/1961245004 conn(0x7fa08c077aa0 msgr2=0x7fa08c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 shutdown_connections 2024-03-11T18:32:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:02.663+0000 7fa090e00700 1 -- 172.21.15.2:0/1961245004 wait complete. 2024-03-11T18:32:03.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:03 smithi165 ceph-mon[36634]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:03.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:03 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1961245004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T18:32:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:03 smithi023 ceph-mon[36406]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:03.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:03 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1961245004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T18:32:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:03 smithi002 ceph-mon[31328]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:03.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:03 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1961245004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-11T18:32:05.411 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:05 smithi002 ceph-mon[31328]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:05.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:05 smithi165 ceph-mon[36634]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:05.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:05 smithi023 ceph-mon[36406]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:05.919 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743464 got 296352743468 for osd.9 2024-03-11T18:32:05.919 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 -- 172.21.15.2:0/334947092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 msgr2=0x7f0dd010a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 --2- 172.21.15.2:0/334947092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd010a210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009990 tx=0x7f0dc0009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 -- 172.21.15.2:0/334947092 shutdown_connections 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 --2- 172.21.15.2:0/334947092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd010a750 0x7f0dd0115110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 --2- 172.21.15.2:0/334947092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd0108560 0x7f0dd0108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 --2- 172.21.15.2:0/334947092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd010a210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 -- 172.21.15.2:0/334947092 >> 172.21.15.2:0/334947092 conn(0x7f0dd00779a0 msgr2=0x7f0dd0106a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:06.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 -- 172.21.15.2:0/334947092 shutdown_connections 2024-03-11T18:32:06.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.132+0000 7f0dd6200700 1 -- 172.21.15.2:0/334947092 wait complete. 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.133+0000 7f0dd6200700 1 Processor -- start 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.133+0000 7f0dd6200700 1 -- start start 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.133+0000 7f0dd6200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd0108560 0x7f0dd019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.133+0000 7f0dd6200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.133+0000 7f0dd6200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd6200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01a3560 con 0x7f0dd0109d90 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd6200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01a36d0 con 0x7f0dd010a750 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd6200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd01a3890 con 0x7f0dd0108560 2024-03-11T18:32:06.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dcfe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd4e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd0108560 0x7f0dd019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56130/0 (socket says 172.21.15.2:56130) 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 -- 172.21.15.2:0/1793830692 learned_addr learned my addr 172.21.15.2:0/1793830692 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 -- 172.21.15.2:0/1793830692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd0108560 msgr2=0x7f0dd019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd0108560 0x7f0dd019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 -- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 msgr2=0x7f0dd019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dd5800700 1 -- 172.21.15.2:0/1793830692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00095f0 con 0x7f0dd010a750 2024-03-11T18:32:06.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.134+0000 7f0dcfe00700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dd5800700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800ef30 tx=0x7f0dc800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8018070 con 0x7f0dd010a750 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd01a3b10 con 0x7f0dd010a750 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0dc800f040 con 0x7f0dd010a750 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800cd50 con 0x7f0dd010a750 2024-03-11T18:32:06.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.135+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd01ab3d0 con 0x7f0dd010a750 2024-03-11T18:32:06.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.136+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd01107f0 con 0x7f0dd010a750 2024-03-11T18:32:06.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.138+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f0dc8003a00 con 0x7f0dd010a750 2024-03-11T18:32:06.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.140+0000 7f0dcd600700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 0x7f0db80a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:06.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.140+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0dc80cd460 con 0x7f0dd010a750 2024-03-11T18:32:06.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.140+0000 7f0dd4e00700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 0x7f0db80a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:06.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.141+0000 7f0dd4e00700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 0x7f0db80a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009c60 tx=0x7f0dc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:06.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.142+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0dc8096200 con 0x7f0dd010a750 2024-03-11T18:32:06.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.382+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f0dd019f9e0 con 0x7f0dd010a750 2024-03-11T18:32:06.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.382+0000 7f0dcd600700 1 -- 172.21.15.2:0/1793830692 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0dc80998c0 con 0x7f0dd010a750 2024-03-11T18:32:06.385 INFO:teuthology.orchestra.run.smithi002.stdout:261993005106 2024-03-11T18:32:06.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.385+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 msgr2=0x7f0db80a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:06.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.385+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 0x7f0db80a4090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009c60 tx=0x7f0dc4009260 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.385+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 msgr2=0x7f0dd01a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.385+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800ef30 tx=0x7f0dc800c3a0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 shutdown_connections 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0db80a1bd0 0x7f0db80a4090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0dd0108560 0x7f0dd019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0dd010a750 0x7f0dd01a2f50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 --2- 172.21.15.2:0/1793830692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0dd0109d90 0x7f0dd019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 >> 172.21.15.2:0/1793830692 conn(0x7f0dd00779a0 msgr2=0x7f0dd0078920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 shutdown_connections 2024-03-11T18:32:06.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:06.386+0000 7f0dd6200700 1 -- 172.21.15.2:0/1793830692 wait complete. 2024-03-11T18:32:07.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:07 smithi002 ceph-mon[31328]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:07.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:07 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1793830692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T18:32:07.574 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:07.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:07 smithi023 ceph-mon[36406]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:07.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:07 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1793830692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T18:32:07.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:07 smithi165 ceph-mon[36634]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:07.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:07 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1793830692' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-11T18:32:08.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411409 got 77309411413 for osd.2 2024-03-11T18:32:08.991 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:09.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:09 smithi165 ceph-mon[36634]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:09.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:09 smithi023 ceph-mon[36406]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:09.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:09 smithi002 ceph-mon[31328]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:10.116 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579938 got 317827579941 for osd.10 2024-03-11T18:32:10.116 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 -- 172.21.15.2:0/1123282263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c0747c0 msgr2=0x7f696c10e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 --2- 172.21.15.2:0/1123282263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c0747c0 0x7f696c10e130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6958009990 tx=0x7f6958009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 -- 172.21.15.2:0/1123282263 shutdown_connections 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 --2- 172.21.15.2:0/1123282263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c073ea0 0x7f696c074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 --2- 172.21.15.2:0/1123282263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c10e670 0x7f696c110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 --2- 172.21.15.2:0/1123282263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c0747c0 0x7f696c10e130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 -- 172.21.15.2:0/1123282263 >> 172.21.15.2:0/1123282263 conn(0x7f696c078da0 msgr2=0x7f696c0791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:10.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 -- 172.21.15.2:0/1123282263 shutdown_connections 2024-03-11T18:32:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.824+0000 7f6973e00700 1 -- 172.21.15.2:0/1123282263 wait complete. 2024-03-11T18:32:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.825+0000 7f6973e00700 1 Processor -- start 2024-03-11T18:32:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.825+0000 7f6973e00700 1 -- start start 2024-03-11T18:32:10.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c073ea0 0x7f696c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 0x7f696c1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 0x7f696c102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1aaa20 con 0x7f696c0747c0 2024-03-11T18:32:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1aab60 con 0x7f696c073ea0 2024-03-11T18:32:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6973e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1aaca0 con 0x7f696c10e670 2024-03-11T18:32:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6970a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 0x7f696c1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6971e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 0x7f696c102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6970a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 0x7f696c1022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43528/0 (socket says 172.21.15.2:43528) 2024-03-11T18:32:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6970a00700 1 -- 172.21.15.2:0/3637897062 learned_addr learned my addr 172.21.15.2:0/3637897062 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.826+0000 7f6971400700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c073ea0 0x7f696c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 -- 172.21.15.2:0/3637897062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c073ea0 msgr2=0x7f696c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c073ea0 0x7f696c101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 -- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 msgr2=0x7f696c1022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 0x7f696c1022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 -- 172.21.15.2:0/3637897062 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69580095f0 con 0x7f696c10e670 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f6971e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 0x7f696c102810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f696400efc0 tx=0x7f696400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964018070 con 0x7f696c10e670 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.827+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f696400f040 con 0x7f696c10e670 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.828+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c1aaf20 con 0x7f696c10e670 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.828+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400cd10 con 0x7f696c10e670 2024-03-11T18:32:10.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.828+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1ab3a0 con 0x7f696c10e670 2024-03-11T18:32:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.829+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c075550 con 0x7f696c10e670 2024-03-11T18:32:10.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.831+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f6964004ad0 con 0x7f696c10e670 2024-03-11T18:32:10.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.832+0000 7f696a000700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 0x7f69540a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.832+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f69640cd360 con 0x7f696c10e670 2024-03-11T18:32:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.832+0000 7f6971400700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 0x7f69540a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.833+0000 7f6971400700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 0x7f69540a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6960009c60 tx=0x7f6960009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:10.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:10.836+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6964096100 con 0x7f696c10e670 2024-03-11T18:32:11.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.068+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f696c114010 con 0x7f696c10e670 2024-03-11T18:32:11.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.069+0000 7f696a000700 1 -- 172.21.15.2:0/3637897062 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f69640997c0 con 0x7f696c10e670 2024-03-11T18:32:11.072 INFO:teuthology.orchestra.run.smithi002.stdout:34359738464 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.072+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 msgr2=0x7f69540a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.072+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 0x7f69540a4010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6960009c60 tx=0x7f6960009260 comp rx=0 tx=0).stop 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.072+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 msgr2=0x7f696c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.072+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 0x7f696c102810 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f696400efc0 tx=0x7f696400c3a0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 shutdown_connections 2024-03-11T18:32:11.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f69540a1b50 0x7f69540a4010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c10e670 0x7f696c102810 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f696c073ea0 0x7f696c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 --2- 172.21.15.2:0/3637897062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f696c0747c0 0x7f696c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 >> 172.21.15.2:0/3637897062 conn(0x7f696c078da0 msgr2=0x7f696c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 shutdown_connections 2024-03-11T18:32:11.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.073+0000 7f6973e00700 1 -- 172.21.15.2:0/3637897062 wait complete. 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 -- 172.21.15.2:0/695155277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe240747c0 msgr2=0x7efe24112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 --2- 172.21.15.2:0/695155277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe240747c0 0x7efe24112900 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efe100098f0 tx=0x7efe10009c00 comp rx=0 tx=0).stop 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 -- 172.21.15.2:0/695155277 shutdown_connections 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 --2- 172.21.15.2:0/695155277 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe24074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 --2- 172.21.15.2:0/695155277 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe24112e40 0x7efe24115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 --2- 172.21.15.2:0/695155277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe240747c0 0x7efe24112900 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 -- 172.21.15.2:0/695155277 >> 172.21.15.2:0/695155277 conn(0x7efe24078dd0 msgr2=0x7efe240791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 -- 172.21.15.2:0/695155277 shutdown_connections 2024-03-11T18:32:11.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.180+0000 7efe29e00700 1 -- 172.21.15.2:0/695155277 wait complete. 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.181+0000 7efe29e00700 1 Processor -- start 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.181+0000 7efe29e00700 1 -- start start 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe241a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe24112e40 0x7efe2419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419ee30 con 0x7efe240747c0 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419efa0 con 0x7efe24112e40 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe29e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2419f160 con 0x7efe24073ea0 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43548/0 (socket says 172.21.15.2:43548) 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 -- 172.21.15.2:0/1067534212 learned_addr learned my addr 172.21.15.2:0/1067534212 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22a00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe241a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:11.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe23400700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe24112e40 0x7efe2419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 -- 172.21.15.2:0/1067534212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 msgr2=0x7efe241a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe241a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 -- 172.21.15.2:0/1067534212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe24112e40 msgr2=0x7efe2419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe24112e40 0x7efe2419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22000700 1 -- 172.21.15.2:0/1067534212 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe10009520 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.182+0000 7efe22a00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe241a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe22000700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efe10005e90 tx=0x7efe1000fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1001c070 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe1000b560 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2419f3e0 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10003c30 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.183+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe241ab530 con 0x7efe240747c0 2024-03-11T18:32:11.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.184+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe240755d0 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.186+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7efe1000b6d0 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.188+0000 7efe28a00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 0x7efe080a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.188+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7efe100ce4f0 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.188+0000 7efe22a00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 0x7efe080a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.189+0000 7efe22a00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 0x7efe080a4010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe14011490 tx=0x7efe14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.192+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efe10097290 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.426+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7efe2419fbe0 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.427+0000 7efe28a00700 1 -- 172.21.15.2:0/1067534212 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7efe1009a950 con 0x7efe240747c0 2024-03-11T18:32:11.586 INFO:teuthology.orchestra.run.smithi002.stdout:352187318306 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.429+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 msgr2=0x7efe080a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.429+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 0x7efe080a4010 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe14011490 tx=0x7efe14009260 comp rx=0 tx=0).stop 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.429+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 msgr2=0x7efe241a4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.429+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efe10005e90 tx=0x7efe1000fa90 comp rx=0 tx=0).stop 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 shutdown_connections 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe080a1b50 0x7efe080a4010 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe24073ea0 0x7efe241a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe24112e40 0x7efe2419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 --2- 172.21.15.2:0/1067534212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe240747c0 0x7efe241a4af0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:11.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 >> 172.21.15.2:0/1067534212 conn(0x7efe24078dd0 msgr2=0x7efe2410e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:11.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 shutdown_connections 2024-03-11T18:32:11.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:11.430+0000 7efe29e00700 1 -- 172.21.15.2:0/1067534212 wait complete. 2024-03-11T18:32:11.588 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:11 smithi002 ceph-mon[31328]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:11.588 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:11 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3637897062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T18:32:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:11 smithi165 ceph-mon[36634]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:11 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3637897062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T18:32:11.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:11 smithi023 ceph-mon[36406]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:11.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:11 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3637897062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T18:32:11.834 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:12 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1067534212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T18:32:12.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:12 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1067534212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T18:32:12.657 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:12 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1067534212' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-11T18:32:13.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005100 got 261993005106 for osd.8 2024-03-11T18:32:13.111 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:13.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:13 smithi002 ceph-mon[31328]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:13.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:13 smithi165 ceph-mon[36634]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:13.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:13 smithi023 ceph-mon[36406]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:13.977 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318300 got 352187318306 for osd.11 2024-03-11T18:32:13.977 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:15 smithi165 ceph-mon[36634]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:15.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:15 smithi023 ceph-mon[36406]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:15.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:15 smithi002 ceph-mon[31328]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:17.108 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738460 got 34359738464 for osd.0 2024-03-11T18:32:17.108 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:17.546 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:17 smithi002 ceph-mon[31328]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:17.547 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2977205288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T18:32:17.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:17 smithi165 ceph-mon[36634]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:17.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2977205288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T18:32:17.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:17 smithi023 ceph-mon[36406]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:17.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2977205288' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 -- 172.21.15.2:0/870196998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534107cc0 msgr2=0x7f15341080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 --2- 172.21.15.2:0/870196998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534107cc0 0x7f15341080a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1528009990 tx=0x7f1528009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 -- 172.21.15.2:0/870196998 shutdown_connections 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 --2- 172.21.15.2:0/870196998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 0x7f153410eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 --2- 172.21.15.2:0/870196998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534107cc0 0x7f15341080a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 --2- 172.21.15.2:0/870196998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534069920 0x7f1534069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 -- 172.21.15.2:0/870196998 >> 172.21.15.2:0/870196998 conn(0x7f1534077aa0 msgr2=0x7f1534077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 -- 172.21.15.2:0/870196998 shutdown_connections 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.880+0000 7f153b600700 1 -- 172.21.15.2:0/870196998 wait complete. 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.881+0000 7f153b600700 1 Processor -- start 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.881+0000 7f153b600700 1 -- start start 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.881+0000 7f153b600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.881+0000 7f153b600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 0x7f153419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f153b600700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534107cc0 0x7f153419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f153b600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153419f290 con 0x7f153406a2c0 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f153b600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153419f400 con 0x7f1534069920 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f153b600700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153419f5c0 con 0x7f1534107cc0 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:17.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46158/0 (socket says 172.21.15.2:46158) 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1533e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 0x7f153419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 -- 172.21.15.2:0/3263402610 learned_addr learned my addr 172.21.15.2:0/3263402610 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1539600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534107cc0 0x7f153419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 -- 172.21.15.2:0/3263402610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534107cc0 msgr2=0x7f153419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534107cc0 0x7f153419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 -- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 msgr2=0x7f153419a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 0x7f153419a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.882+0000 7f1538c00700 1 -- 172.21.15.2:0/3263402610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280095f0 con 0x7f1534069920 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f1538c00700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152801d070 con 0x7f1534069920 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153419f840 con 0x7f1534069920 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f152800b610 con 0x7f1534069920 2024-03-11T18:32:17.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280173f0 con 0x7f1534069920 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.883+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15341a7100 con 0x7f1534069920 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.884+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1534103370 con 0x7f1534069920 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.887+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f1528022670 con 0x7f1534069920 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.888+0000 7f1531600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 0x7f151c0a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.888+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f15280cec90 con 0x7f1534069920 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.888+0000 7f1533e00700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 0x7f151c0a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.889+0000 7f1533e00700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 0x7f151c0a4090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1534102c00 tx=0x7f1520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:17.890+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1528097ac0 con 0x7f1534069920 2024-03-11T18:32:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.120+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1534068930 con 0x7f1534069920 2024-03-11T18:32:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.120+0000 7f1531600700 1 -- 172.21.15.2:0/3263402610 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f152809b180 con 0x7f1534069920 2024-03-11T18:32:18.122 INFO:teuthology.orchestra.run.smithi002.stdout:55834574940 2024-03-11T18:32:18.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.123+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 msgr2=0x7f151c0a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.123+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 0x7f151c0a4090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1534102c00 tx=0x7f1520009260 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.123+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 msgr2=0x7f153419a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.123+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f1528005740 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 shutdown_connections 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f151c0a1bd0 0x7f151c0a4090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1534107cc0 0x7f153419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1534069920 0x7f153419a3b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 --2- 172.21.15.2:0/3263402610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f153406a2c0 0x7f153419a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 >> 172.21.15.2:0/3263402610 conn(0x7f1534077aa0 msgr2=0x7f153410ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 shutdown_connections 2024-03-11T18:32:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.124+0000 7f153b600700 1 -- 172.21.15.2:0/3263402610 wait complete. 2024-03-11T18:32:18.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.188+0000 7f9056a00700 1 -- 172.21.15.2:0/1562249548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905006a2c0 msgr2=0x7f905010eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.188+0000 7f9056a00700 1 --2- 172.21.15.2:0/1562249548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905006a2c0 0x7f905010eb00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f904800b0a0 tx=0x7f904800b3b0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 -- 172.21.15.2:0/1562249548 shutdown_connections 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 --2- 172.21.15.2:0/1562249548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f905006a2c0 0x7f905010eb00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 --2- 172.21.15.2:0/1562249548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050107cc0 0x7f90501080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 --2- 172.21.15.2:0/1562249548 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9050069920 0x7f9050069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 -- 172.21.15.2:0/1562249548 >> 172.21.15.2:0/1562249548 conn(0x7f9050077aa0 msgr2=0x7f9050077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.189+0000 7f9056a00700 1 -- 172.21.15.2:0/1562249548 shutdown_connections 2024-03-11T18:32:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.190+0000 7f9056a00700 1 -- 172.21.15.2:0/1562249548 wait complete. 2024-03-11T18:32:18.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 Processor -- start 2024-03-11T18:32:18.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 -- start start 2024-03-11T18:32:18.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f905006a2c0 0x7f9050198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9050107cc0 0x7f905019cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905019d130 con 0x7f9050069920 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905019d2a0 con 0x7f9050107cc0 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f9056a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905019d460 con 0x7f905006a2c0 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.191+0000 7f904fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54306/0 (socket says 172.21.15.2:54306) 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 -- 172.21.15.2:0/511247878 learned_addr learned my addr 172.21.15.2:0/511247878 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:18.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f9054a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9050107cc0 0x7f905019cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904f400700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f905006a2c0 0x7f9050198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 -- 172.21.15.2:0/511247878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f905006a2c0 msgr2=0x7f9050198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f905006a2c0 0x7f9050198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 -- 172.21.15.2:0/511247878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9050107cc0 msgr2=0x7f905019cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9050107cc0 0x7f905019cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 -- 172.21.15.2:0/511247878 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048009d00 con 0x7f9050069920 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.192+0000 7f904fe00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f904000e9a0 tx=0x7f904000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.193+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040009790 con 0x7f9050069920 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.193+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9040010430 con 0x7f9050069920 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.193+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905019d740 con 0x7f9050069920 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.193+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9040018680 con 0x7f9050069920 2024-03-11T18:32:18.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.193+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90501a4fa0 con 0x7f9050069920 2024-03-11T18:32:18.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.194+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905010a1e0 con 0x7f9050069920 2024-03-11T18:32:18.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.196+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f9040010910 con 0x7f9050069920 2024-03-11T18:32:18.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.199+0000 7f904cc00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 0x7f90340a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.199+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f9040014070 con 0x7f9050069920 2024-03-11T18:32:18.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.199+0000 7f904f400700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 0x7f90340a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.200+0000 7f904f400700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 0x7f90340a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9050078010 tx=0x7f903c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:18.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.205+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9040052d20 con 0x7f9050069920 2024-03-11T18:32:18.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.439+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9050111ca0 con 0x7f9050069920 2024-03-11T18:32:18.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3263402610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T18:32:18.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.440+0000 7f904cc00700 1 -- 172.21.15.2:0/511247878 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f90400967f0 con 0x7f9050069920 2024-03-11T18:32:18.442 INFO:teuthology.orchestra.run.smithi002.stdout:206158430272 2024-03-11T18:32:18.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 msgr2=0x7f90340a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 0x7f90340a4010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9050078010 tx=0x7f903c009260 comp rx=0 tx=0).stop 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 msgr2=0x7f9050198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f904000e9a0 tx=0x7f904000ecb0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 shutdown_connections 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9050107cc0 0x7f905019cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9050069920 0x7f9050198250 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f90340a1b50 0x7f90340a4010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 --2- 172.21.15.2:0/511247878 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f905006a2c0 0x7f9050198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 >> 172.21.15.2:0/511247878 conn(0x7f9050077aa0 msgr2=0x7f90500ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 shutdown_connections 2024-03-11T18:32:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:18.443+0000 7f9056a00700 1 -- 172.21.15.2:0/511247878 wait complete. 2024-03-11T18:32:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:18 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3263402610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T18:32:18.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:18 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3263402610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T18:32:18.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.219+0000 7efd9f200700 1 -- 172.21.15.2:0/2500928892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98073ea0 msgr2=0x7efd98074280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.219+0000 7efd9f200700 1 --2- 172.21.15.2:0/2500928892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98073ea0 0x7efd98074280 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efd88009990 tx=0x7efd88009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 -- 172.21.15.2:0/2500928892 shutdown_connections 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 --2- 172.21.15.2:0/2500928892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98112e40 0x7efd98115230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 --2- 172.21.15.2:0/2500928892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98073ea0 0x7efd98074280 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 --2- 172.21.15.2:0/2500928892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 0x7efd98112900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 -- 172.21.15.2:0/2500928892 >> 172.21.15.2:0/2500928892 conn(0x7efd98078dd0 msgr2=0x7efd980791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 -- 172.21.15.2:0/2500928892 shutdown_connections 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.220+0000 7efd9f200700 1 -- 172.21.15.2:0/2500928892 wait complete. 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.222+0000 7efd9f200700 1 Processor -- start 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.222+0000 7efd9f200700 1 -- start start 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.222+0000 7efd9f200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9f200700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 0x7efd981a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9f200700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 0x7efd9819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9f200700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9819ec70 con 0x7efd98112e40 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9f200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9819edb0 con 0x7efd98073ea0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9f200700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9819ef70 con 0x7efd980747c0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:57358/0 (socket says 172.21.15.2:57358) 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 -- 172.21.15.2:0/2977205288 learned_addr learned my addr 172.21.15.2:0/2977205288 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97400700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 0x7efd981a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9d200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 0x7efd9819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 -- 172.21.15.2:0/2977205288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 msgr2=0x7efd981a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 0x7efd981a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 -- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 msgr2=0x7efd9819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 0x7efd9819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd97e00700 1 -- 172.21.15.2:0/2977205288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd880095f0 con 0x7efd98073ea0 2024-03-11T18:32:18.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.223+0000 7efd9d200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 0x7efd9819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd97e00700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd8800f690 tx=0x7efd8800f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8801d070 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd88003680 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9819f1f0 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd880173f0 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.224+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd981ab4f0 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.225+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98068930 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.227+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7efd88022470 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.230+0000 7efd94c00700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 0x7efd7c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.230+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7efd880ceaa0 con 0x7efd98073ea0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.230+0000 7efd97400700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 0x7efd7c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.231+0000 7efd97400700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 0x7efd7c0a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd98114d20 tx=0x7efd8400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:18.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.231+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd8805dd20 con 0x7efd98073ea0 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.482+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7efd98118390 con 0x7efd98073ea0 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.483+0000 7efd94c00700 1 -- 172.21.15.2:0/2977205288 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efd88097840 con 0x7efd98073ea0 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stdout:171798691910 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 msgr2=0x7efd7c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 0x7efd7c0a4010 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efd98114d20 tx=0x7efd8400b410 comp rx=0 tx=0).stop 2024-03-11T18:32:18.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 msgr2=0x7efd981a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efd8800f690 tx=0x7efd8800f6c0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 shutdown_connections 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd98112e40 0x7efd9819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd98073ea0 0x7efd981a4450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efd7c0a1b50 0x7efd7c0a4010 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 --2- 172.21.15.2:0/2977205288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd980747c0 0x7efd981a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 >> 172.21.15.2:0/2977205288 conn(0x7efd98078dd0 msgr2=0x7efd98111c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 shutdown_connections 2024-03-11T18:32:18.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:16.486+0000 7efd9f200700 1 -- 172.21.15.2:0/2977205288 wait complete. 2024-03-11T18:32:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 -- 172.21.15.2:0/3574858041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 msgr2=0x7f645c06acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 --2- 172.21.15.2:0/3574858041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c06acd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6448009920 tx=0x7f6448009c30 comp rx=0 tx=0).stop 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 -- 172.21.15.2:0/3574858041 shutdown_connections 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 --2- 172.21.15.2:0/3574858041 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 0x7f645c104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 --2- 172.21.15.2:0/3574858041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c06a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 --2- 172.21.15.2:0/3574858041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c06acd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.388+0000 7f6460c00700 1 -- 172.21.15.2:0/3574858041 >> 172.21.15.2:0/3574858041 conn(0x7f645c077ae0 msgr2=0x7f645c077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.389+0000 7f6460c00700 1 -- 172.21.15.2:0/3574858041 shutdown_connections 2024-03-11T18:32:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.389+0000 7f6460c00700 1 -- 172.21.15.2:0/3574858041 wait complete. 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 Processor -- start 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 -- start start 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c1a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 0x7f645c1a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c19ede0 con 0x7f645c106b80 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c19ef50 con 0x7f645c06a850 2024-03-11T18:32:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.390+0000 7f6460c00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c19f110 con 0x7f645c1046a0 2024-03-11T18:32:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.391+0000 7f645a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.391+0000 7f6459e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c1a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.391+0000 7f645a800700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54324/0 (socket says 172.21.15.2:54324) 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.391+0000 7f645a800700 1 -- 172.21.15.2:0/1678527224 learned_addr learned my addr 172.21.15.2:0/1678527224 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.391+0000 7f6459400700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 0x7f645c1a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 -- 172.21.15.2:0/1678527224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 msgr2=0x7f645c1a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 0x7f645c1a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 -- 172.21.15.2:0/1678527224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 msgr2=0x7f645c1a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c1a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 -- 172.21.15.2:0/1678527224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448009520 con 0x7f645c106b80 2024-03-11T18:32:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f6459e00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c1a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f645a800700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f645400efc0 tx=0x7f645400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6454018070 con 0x7f645c106b80 2024-03-11T18:32:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f645400f040 con 0x7f645c106b80 2024-03-11T18:32:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.392+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c19f390 con 0x7f645c106b80 2024-03-11T18:32:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.393+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645400ce20 con 0x7f645c106b80 2024-03-11T18:32:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.393+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c1ab4e0 con 0x7f645c106b80 2024-03-11T18:32:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.394+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c101180 con 0x7f645c106b80 2024-03-11T18:32:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.396+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f6454003a00 con 0x7f645c106b80 2024-03-11T18:32:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.397+0000 7f6452a00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 0x7f64400a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.398+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f64540cd3c0 con 0x7f645c106b80 2024-03-11T18:32:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.398+0000 7f6459e00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 0x7f64400a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.399+0000 7f6459e00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 0x7f64400a4010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f644c005fd0 tx=0x7f644c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:19.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.403+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6454096160 con 0x7f645c106b80 2024-03-11T18:32:19.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.638+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f645c118340 con 0x7f645c106b80 2024-03-11T18:32:19.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:19 smithi002 ceph-mon[31328]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:19.640 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/511247878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T18:32:19.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.639+0000 7f6452a00700 1 -- 172.21.15.2:0/1678527224 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6454099820 con 0x7f645c106b80 2024-03-11T18:32:19.641 INFO:teuthology.orchestra.run.smithi002.stdout:150323855436 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 msgr2=0x7f64400a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 0x7f64400a4010 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f644c005fd0 tx=0x7f644c009310 comp rx=0 tx=0).stop 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 msgr2=0x7f645c19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f645400efc0 tx=0x7f645400c450 comp rx=0 tx=0).stop 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 shutdown_connections 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f645c106b80 0x7f645c19e630 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f645c06a850 0x7f645c1a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f64400a1b50 0x7f64400a4010 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 --2- 172.21.15.2:0/1678527224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f645c1046a0 0x7f645c1a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:19.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 >> 172.21.15.2:0/1678527224 conn(0x7f645c077ae0 msgr2=0x7f645c103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:19.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 shutdown_connections 2024-03-11T18:32:19.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:19.642+0000 7f6460c00700 1 -- 172.21.15.2:0/1678527224 wait complete. 2024-03-11T18:32:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:19 smithi023 ceph-mon[36406]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:19.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:19 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/511247878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T18:32:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:19 smithi165 ceph-mon[36634]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:19 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/511247878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T18:32:19.819 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574940 for osd.1 2024-03-11T18:32:19.820 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:20 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1678527224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T18:32:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:20 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:20 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1678527224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T18:32:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:20 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:20.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:20 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1678527224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T18:32:20.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:20.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:20 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:21.333 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691903 got 171798691910 for osd.5 2024-03-11T18:32:21.333 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:21.443 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430263 got 206158430272 for osd.6 2024-03-11T18:32:21.443 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:21.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855430 got 150323855436 for osd.4 2024-03-11T18:32:21.806 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:21.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.987+0000 7fd2a3400700 1 -- 172.21.15.2:0/1590085257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c101eb0 msgr2=0x7fd29c102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:21.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.987+0000 7fd2a3400700 1 --2- 172.21.15.2:0/1590085257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c101eb0 0x7fd29c102290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd290009990 tx=0x7fd290009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:21 smithi002 ceph-mon[31328]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 -- 172.21.15.2:0/1590085257 shutdown_connections 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 --2- 172.21.15.2:0/1590085257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c106f90 0x7fd29c10eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 --2- 172.21.15.2:0/1590085257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c101eb0 0x7fd29c102290 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 --2- 172.21.15.2:0/1590085257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 -- 172.21.15.2:0/1590085257 >> 172.21.15.2:0/1590085257 conn(0x7fd29c0fd720 msgr2=0x7fd29c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 -- 172.21.15.2:0/1590085257 shutdown_connections 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.988+0000 7fd2a3400700 1 -- 172.21.15.2:0/1590085257 wait complete. 2024-03-11T18:32:21.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.989+0000 7fd2a3400700 1 Processor -- start 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.989+0000 7fd2a3400700 1 -- start start 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 0x7fd29c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c106f90 0x7fd29c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c19f2e0 con 0x7fd29c106f90 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c19f450 con 0x7fd29c102860 2024-03-11T18:32:21.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a3400700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c19f610 con 0x7fd29c101eb0 2024-03-11T18:32:21.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a0a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 0x7fd29c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:21.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a0a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 0x7fd29c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58334/0 (socket says 172.21.15.2:58334) 2024-03-11T18:32:21.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.990+0000 7fd2a0a00700 1 -- 172.21.15.2:0/761460566 learned_addr learned my addr 172.21.15.2:0/761460566 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:21.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd2a1400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c106f90 0x7fd29c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:21.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 -- 172.21.15.2:0/761460566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 msgr2=0x7fd29c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 0x7fd29c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 -- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c106f90 msgr2=0x7fd29c19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c106f90 0x7fd29c19ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.991+0000 7fd29be00700 1 -- 172.21.15.2:0/761460566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900095f0 con 0x7fd29c102860 2024-03-11T18:32:21.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd29be00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c19a940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd29c06a700 tx=0x7fd28800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800f890 con 0x7fd29c102860 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd288010460 con 0x7fd29c102860 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c19f8f0 con 0x7fd29c102860 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800e610 con 0x7fd29c102860 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.992+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c1a7150 con 0x7fd29c102860 2024-03-11T18:32:21.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.993+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c068930 con 0x7fd29c102860 2024-03-11T18:32:21.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.995+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7fd288010920 con 0x7fd29c102860 2024-03-11T18:32:21.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.997+0000 7fd299600700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 0x7fd2840a4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:21.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.997+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fd2880cd4e0 con 0x7fd29c102860 2024-03-11T18:32:21.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.997+0000 7fd2a0a00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 0x7fd2840a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:21.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:21.998+0000 7fd2a0a00700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 0x7fd2840a4090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd290006010 tx=0x7fd29000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:22.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.001+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd2880961f0 con 0x7fd29c102860 2024-03-11T18:32:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:21 smithi165 ceph-mon[36634]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:22.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:21 smithi023 ceph-mon[36406]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:22.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.231+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd29c111d40 con 0x7fd29c102860 2024-03-11T18:32:22.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.232+0000 7fd299600700 1 -- 172.21.15.2:0/761460566 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd2880998b0 con 0x7fd29c102860 2024-03-11T18:32:22.233 INFO:teuthology.orchestra.run.smithi002.stdout:227633266747 2024-03-11T18:32:22.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 msgr2=0x7fd2840a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:22.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 0x7fd2840a4090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd290006010 tx=0x7fd29000b260 comp rx=0 tx=0).stop 2024-03-11T18:32:22.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 msgr2=0x7fd29c19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c19a940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd29c06a700 tx=0x7fd28800dc30 comp rx=0 tx=0).stop 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 shutdown_connections 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd2840a1bd0 0x7fd2840a4090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.234+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd29c101eb0 0x7fd29c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.235+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd29c106f90 0x7fd29c19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.235+0000 7fd2a3400700 1 --2- 172.21.15.2:0/761460566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd29c102860 0x7fd29c19a940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.235+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 >> 172.21.15.2:0/761460566 conn(0x7fd29c0fd720 msgr2=0x7fd29c0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.235+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 shutdown_connections 2024-03-11T18:32:22.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:22.235+0000 7fd2a3400700 1 -- 172.21.15.2:0/761460566 wait complete. 2024-03-11T18:32:23.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:32:23.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:32:23.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:22 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:23.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:22 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/761460566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T18:32:23.061 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266741 got 227633266747 for osd.7 2024-03-11T18:32:23.062 DEBUG:teuthology.parallel:result is None 2024-03-11T18:32:23.062 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T18:32:23.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T18:32:23.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:32:23.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:32:23.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:22 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:23.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:22 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/761460566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T18:32:23.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:32:23.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:32:23.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:22 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:32:23.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:22 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/761460566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T18:32:23.554 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:24.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:23 smithi002 ceph-mon[31328]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:24.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:23 smithi165 ceph-mon[36634]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:24.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:23 smithi023 ceph-mon[36406]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.785+0000 7f3f7c400700 1 -- 172.21.15.2:0/452782173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f7406a2c0 msgr2=0x7f3f7410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.785+0000 7f3f7c400700 1 --2- 172.21.15.2:0/452782173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f7406a2c0 0x7f3f7410eb00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009fc0 tx=0x7f3f6c00b330 comp rx=0 tx=0).stop 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 -- 172.21.15.2:0/452782173 shutdown_connections 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 --2- 172.21.15.2:0/452782173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f7406a2c0 0x7f3f7410eb00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 --2- 172.21.15.2:0/452782173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 0x7f3f741080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 --2- 172.21.15.2:0/452782173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f74069920 0x7f3f74069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 -- 172.21.15.2:0/452782173 >> 172.21.15.2:0/452782173 conn(0x7f3f74077aa0 msgr2=0x7f3f74077eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:25.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 -- 172.21.15.2:0/452782173 shutdown_connections 2024-03-11T18:32:25.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.786+0000 7f3f7c400700 1 -- 172.21.15.2:0/452782173 wait complete. 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.787+0000 7f3f7c400700 1 Processor -- start 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.787+0000 7f3f7c400700 1 -- start start 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 0x7f3f7419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 0x7f3f7419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7419f390 con 0x7f3f74069920 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7419f500 con 0x7f3f74107cc0 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7c400700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7419f6c0 con 0x7f3f7406a2c0 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f79a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f79a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54364/0 (socket says 172.21.15.2:54364) 2024-03-11T18:32:25.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f79a00700 1 -- 172.21.15.2:0/4108961663 learned_addr learned my addr 172.21.15.2:0/4108961663 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f79000700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 0x7f3f7419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.788+0000 7f3f7a400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 0x7f3f7419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 -- 172.21.15.2:0/4108961663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 msgr2=0x7f3f7419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 0x7f3f7419a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 -- 172.21.15.2:0/4108961663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 msgr2=0x7f3f7419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 0x7f3f7419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 -- 172.21.15.2:0/4108961663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680095f0 con 0x7f3f74069920 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79000700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 0x7f3f7419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.789+0000 7f3f79a00700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800ec10 tx=0x7f3f6800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.790+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6800cc90 con 0x7f3f74069920 2024-03-11T18:32:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.790+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f68004500 con 0x7f3f74069920 2024-03-11T18:32:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.790+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c009c40 con 0x7f3f74069920 2024-03-11T18:32:25.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.790+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68010690 con 0x7f3f74069920 2024-03-11T18:32:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.790+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f741a7030 con 0x7f3f74069920 2024-03-11T18:32:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.791+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f74068930 con 0x7f3f74069920 2024-03-11T18:32:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.794+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 136329+0+0 (secure 0 0 0) 0x7f3f68003710 con 0x7f3f74069920 2024-03-11T18:32:25.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.795+0000 7f3f72000700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 0x7f3f5c0a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.795+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f3f68014070 con 0x7f3f74069920 2024-03-11T18:32:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.795+0000 7f3f79000700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 0x7f3f5c0a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.796+0000 7f3f79000700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 0x7f3f5c0a4010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3f74102b70 tx=0x7f3f60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:25.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:25.798+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f68096b90 con 0x7f3f74069920 2024-03-11T18:32:26.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.022+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f74111cf0 con 0x7f3f5c0a1b50 2024-03-11T18:32:26.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.043+0000 7f3f72000700 1 -- 172.21.15.2:0/4108961663 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340368 (secure 0 0 0) 0x7f3f74111cf0 con 0x7f3f5c0a1b50 2024-03-11T18:32:26.046 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:26.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 msgr2=0x7f3f5c0a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:26.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 0x7f3f5c0a4010 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3f74102b70 tx=0x7f3f60009260 comp rx=0 tx=0).stop 2024-03-11T18:32:26.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 msgr2=0x7f3f7419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:26.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f6800ec10 tx=0x7f3f6800ef20 comp rx=0 tx=0).stop 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 shutdown_connections 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f74107cc0 0x7f3f7419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f74069920 0x7f3f7419a4b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3f5c0a1b50 0x7f3f5c0a4010 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 --2- 172.21.15.2:0/4108961663 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f7406a2c0 0x7f3f7419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 >> 172.21.15.2:0/4108961663 conn(0x7f3f74077aa0 msgr2=0x7f3f7410ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 shutdown_connections 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:26.053+0000 7f3f7c400700 1 -- 172.21.15.2:0/4108961663 wait complete. 2024-03-11T18:32:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T18:32:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:25 smithi165 ceph-mon[36634]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:26.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:25 smithi023 ceph-mon[36406]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:25 smithi002 ceph-mon[31328]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:26.885 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-03-11T18:32:24.248587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658350,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":633,"num_read_kb":441,"num_write":349,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1181866,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367668,"kb_used_data":14288,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574156,"statfs":{"total":1151940427776,"available":1151563935744,"internally_reserved":0,"allocated":14630912,"data_stored":8040272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004825"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586747+0000","last_change":"2024-03-11T18:30:28.060909+0000","last_active":"2024-03-11T18:31:29.586747+0000","last_peered":"2024-03-11T18:31:29.586747+0000","last_clean":"2024-03-11T18:31:29.586747+0000","last_became_active":"2024-03-11T18:30:28.060554+0000","last_became_peered":"2024-03-11T18:30:28.060554+0000","last_unstale":"2024-03-11T18:31:29.586747+0000","last_undegraded":"2024-03-11T18:31:29.586747+0000","last_fullsized":"2024-03-11T18:31:29.586747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:33.609820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591731+0000","last_change":"2024-03-11T18:30:28.055136+0000","last_active":"2024-03-11T18:31:29.591731+0000","last_peered":"2024-03-11T18:31:29.591731+0000","last_clean":"2024-03-11T18:31:29.591731+0000","last_became_active":"2024-03-11T18:30:28.054785+0000","last_became_peered":"2024-03-11T18:30:28.054785+0000","last_unstale":"2024-03-11T18:31:29.591731+0000","last_undegraded":"2024-03-11T18:31:29.591731+0000","last_fullsized":"2024-03-11T18:31:29.591731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:12.702850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590006+0000","last_change":"2024-03-11T18:30:28.051593+0000","last_active":"2024-03-11T18:31:29.590006+0000","last_peered":"2024-03-11T18:31:29.590006+0000","last_clean":"2024-03-11T18:31:29.590006+0000","last_became_active":"2024-03-11T18:30:28.049344+0000","last_became_peered":"2024-03-11T18:30:28.049344+0000","last_unstale":"2024-03-11T18:31:29.590006+0000","last_undegraded":"2024-03-11T18:31:29.590006+0000","last_fullsized":"2024-03-11T18:31:29.590006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:06.433202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585318+0000","last_change":"2024-03-11T18:30:28.036335+0000","last_active":"2024-03-11T18:31:29.585318+0000","last_peered":"2024-03-11T18:31:29.585318+0000","last_clean":"2024-03-11T18:31:29.585318+0000","last_became_active":"2024-03-11T18:30:28.035728+0000","last_became_peered":"2024-03-11T18:30:28.035728+0000","last_unstale":"2024-03-11T18:31:29.585318+0000","last_undegraded":"2024-03-11T18:31:29.585318+0000","last_fullsized":"2024-03-11T18:31:29.585318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:10.728266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610339+0000","last_change":"2024-03-11T18:30:28.050560+0000","last_active":"2024-03-11T18:31:29.610339+0000","last_peered":"2024-03-11T18:31:29.610339+0000","last_clean":"2024-03-11T18:31:29.610339+0000","last_became_active":"2024-03-11T18:30:28.050273+0000","last_became_peered":"2024-03-11T18:30:28.050273+0000","last_unstale":"2024-03-11T18:31:29.610339+0000","last_undegraded":"2024-03-11T18:31:29.610339+0000","last_fullsized":"2024-03-11T18:31:29.610339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:56.780575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609873+0000","last_change":"2024-03-11T18:30:28.056879+0000","last_active":"2024-03-11T18:31:29.609873+0000","last_peered":"2024-03-11T18:31:29.609873+0000","last_clean":"2024-03-11T18:31:29.609873+0000","last_became_active":"2024-03-11T18:30:28.056435+0000","last_became_peered":"2024-03-11T18:30:28.056435+0000","last_unstale":"2024-03-11T18:31:29.609873+0000","last_undegraded":"2024-03-11T18:31:29.609873+0000","last_fullsized":"2024-03-11T18:31:29.609873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:04.595503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584088+0000","last_change":"2024-03-11T18:30:28.044355+0000","last_active":"2024-03-11T18:31:29.584088+0000","last_peered":"2024-03-11T18:31:29.584088+0000","last_clean":"2024-03-11T18:31:29.584088+0000","last_became_active":"2024-03-11T18:30:28.043875+0000","last_became_peered":"2024-03-11T18:30:28.043875+0000","last_unstale":"2024-03-11T18:31:29.584088+0000","last_undegraded":"2024-03-11T18:31:29.584088+0000","last_fullsized":"2024-03-11T18:31:29.584088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:25.924359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609458+0000","last_change":"2024-03-11T18:30:28.055633+0000","last_active":"2024-03-11T18:31:29.609458+0000","last_peered":"2024-03-11T18:31:29.609458+0000","last_clean":"2024-03-11T18:31:29.609458+0000","last_became_active":"2024-03-11T18:30:28.055158+0000","last_became_peered":"2024-03-11T18:30:28.055158+0000","last_unstale":"2024-03-11T18:31:29.609458+0000","last_undegraded":"2024-03-11T18:31:29.609458+0000","last_fullsized":"2024-03-11T18:31:29.609458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:21:45.379551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609345+0000","last_change":"2024-03-11T18:30:28.037306+0000","last_active":"2024-03-11T18:31:29.609345+0000","last_peered":"2024-03-11T18:31:29.609345+0000","last_clean":"2024-03-11T18:31:29.609345+0000","last_became_active":"2024-03-11T18:30:28.036930+0000","last_became_peered":"2024-03-11T18:30:28.036930+0000","last_unstale":"2024-03-11T18:31:29.609345+0000","last_undegraded":"2024-03-11T18:31:29.609345+0000","last_fullsized":"2024-03-11T18:31:29.609345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610002+0000","last_change":"2024-03-11T18:30:28.038914+0000","last_active":"2024-03-11T18:31:29.610002+0000","last_peered":"2024-03-11T18:31:29.610002+0000","last_clean":"2024-03-11T18:31:29.610002+0000","last_became_active":"2024-03-11T18:30:28.038266+0000","last_became_peered":"2024-03-11T18:30:28.038266+0000","last_unstale":"2024-03-11T18:31:29.610002+0000","last_undegraded":"2024-03-11T18:31:29.610002+0000","last_fullsized":"2024-03-11T18:31:29.610002+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:59.136860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584650+0000","last_change":"2024-03-11T18:30:28.041833+0000","last_active":"2024-03-11T18:31:29.584650+0000","last_peered":"2024-03-11T18:31:29.584650+0000","last_clean":"2024-03-11T18:31:29.584650+0000","last_became_active":"2024-03-11T18:30:28.041307+0000","last_became_peered":"2024-03-11T18:30:28.041307+0000","last_unstale":"2024-03-11T18:31:29.584650+0000","last_undegraded":"2024-03-11T18:31:29.584650+0000","last_fullsized":"2024-03-11T18:31:29.584650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:35.803202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585230+0000","last_change":"2024-03-11T18:30:28.037646+0000","last_active":"2024-03-11T18:31:29.585230+0000","last_peered":"2024-03-11T18:31:29.585230+0000","last_clean":"2024-03-11T18:31:29.585230+0000","last_became_active":"2024-03-11T18:30:28.037272+0000","last_became_peered":"2024-03-11T18:30:28.037272+0000","last_unstale":"2024-03-11T18:31:29.585230+0000","last_undegraded":"2024-03-11T18:31:29.585230+0000","last_fullsized":"2024-03-11T18:31:29.585230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:50.594666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580926+0000","last_change":"2024-03-11T18:30:28.056917+0000","last_active":"2024-03-11T18:31:29.580926+0000","last_peered":"2024-03-11T18:31:29.580926+0000","last_clean":"2024-03-11T18:31:29.580926+0000","last_became_active":"2024-03-11T18:30:28.056496+0000","last_became_peered":"2024-03-11T18:30:28.056496+0000","last_unstale":"2024-03-11T18:31:29.580926+0000","last_undegraded":"2024-03-11T18:31:29.580926+0000","last_fullsized":"2024-03-11T18:31:29.580926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:44.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594231+0000","last_change":"2024-03-11T18:30:28.055180+0000","last_active":"2024-03-11T18:31:29.594231+0000","last_peered":"2024-03-11T18:31:29.594231+0000","last_clean":"2024-03-11T18:31:29.594231+0000","last_became_active":"2024-03-11T18:30:28.054807+0000","last_became_peered":"2024-03-11T18:30:28.054807+0000","last_unstale":"2024-03-11T18:31:29.594231+0000","last_undegraded":"2024-03-11T18:31:29.594231+0000","last_fullsized":"2024-03-11T18:31:29.594231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:31.671461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590883+0000","last_change":"2024-03-11T18:30:28.065015+0000","last_active":"2024-03-11T18:31:29.590883+0000","last_peered":"2024-03-11T18:31:29.590883+0000","last_clean":"2024-03-11T18:31:29.590883+0000","last_became_active":"2024-03-11T18:30:28.061131+0000","last_became_peered":"2024-03-11T18:30:28.061131+0000","last_unstale":"2024-03-11T18:31:29.590883+0000","last_undegraded":"2024-03-11T18:31:29.590883+0000","last_fullsized":"2024-03-11T18:31:29.590883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:35.314562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608751+0000","last_change":"2024-03-11T18:30:28.054388+0000","last_active":"2024-03-11T18:31:29.608751+0000","last_peered":"2024-03-11T18:31:29.608751+0000","last_clean":"2024-03-11T18:31:29.608751+0000","last_became_active":"2024-03-11T18:30:28.054036+0000","last_became_peered":"2024-03-11T18:30:28.054036+0000","last_unstale":"2024-03-11T18:31:29.608751+0000","last_undegraded":"2024-03-11T18:31:29.608751+0000","last_fullsized":"2024-03-11T18:31:29.608751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:45:21.476944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586710+0000","last_change":"2024-03-11T18:30:28.061935+0000","last_active":"2024-03-11T18:31:29.586710+0000","last_peered":"2024-03-11T18:31:29.586710+0000","last_clean":"2024-03-11T18:31:29.586710+0000","last_became_active":"2024-03-11T18:30:28.061558+0000","last_became_peered":"2024-03-11T18:30:28.061558+0000","last_unstale":"2024-03-11T18:31:29.586710+0000","last_undegraded":"2024-03-11T18:31:29.586710+0000","last_fullsized":"2024-03-11T18:31:29.586710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:01.728956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594476+0000","last_change":"2024-03-11T18:30:28.058372+0000","last_active":"2024-03-11T18:31:29.594476+0000","last_peered":"2024-03-11T18:31:29.594476+0000","last_clean":"2024-03-11T18:31:29.594476+0000","last_became_active":"2024-03-11T18:30:28.058028+0000","last_became_peered":"2024-03-11T18:30:28.058028+0000","last_unstale":"2024-03-11T18:31:29.594476+0000","last_undegraded":"2024-03-11T18:31:29.594476+0000","last_fullsized":"2024-03-11T18:31:29.594476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:20:15.645368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580740+0000","last_change":"2024-03-11T18:30:28.054845+0000","last_active":"2024-03-11T18:31:29.580740+0000","last_peered":"2024-03-11T18:31:29.580740+0000","last_clean":"2024-03-11T18:31:29.580740+0000","last_became_active":"2024-03-11T18:30:28.054497+0000","last_became_peered":"2024-03-11T18:30:28.054497+0000","last_unstale":"2024-03-11T18:31:29.580740+0000","last_undegraded":"2024-03-11T18:31:29.580740+0000","last_fullsized":"2024-03-11T18:31:29.580740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:52.100045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580986+0000","last_change":"2024-03-11T18:30:32.322103+0000","last_active":"2024-03-11T18:31:29.580986+0000","last_peered":"2024-03-11T18:31:29.580986+0000","last_clean":"2024-03-11T18:31:29.580986+0000","last_became_active":"2024-03-11T18:30:32.321422+0000","last_became_peered":"2024-03-11T18:30:32.321422+0000","last_unstale":"2024-03-11T18:31:29.580986+0000","last_undegraded":"2024-03-11T18:31:29.580986+0000","last_fullsized":"2024-03-11T18:31:29.580986+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:56.946318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.596373+0000","last_change":"2024-03-11T18:30:27.283463+0000","last_active":"2024-03-11T18:31:29.596373+0000","last_peered":"2024-03-11T18:31:29.596373+0000","last_clean":"2024-03-11T18:31:29.596373+0000","last_became_active":"2024-03-11T18:30:27.279100+0000","last_became_peered":"2024-03-11T18:30:27.279100+0000","last_unstale":"2024-03-11T18:31:29.596373+0000","last_undegraded":"2024-03-11T18:31:29.596373+0000","last_fullsized":"2024-03-11T18:31:29.596373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:38.096395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610504+0000","last_change":"2024-03-11T18:30:28.051288+0000","last_active":"2024-03-11T18:31:29.610504+0000","last_peered":"2024-03-11T18:31:29.610504+0000","last_clean":"2024-03-11T18:31:29.610504+0000","last_became_active":"2024-03-11T18:30:28.050726+0000","last_became_peered":"2024-03-11T18:30:28.050726+0000","last_unstale":"2024-03-11T18:31:29.610504+0000","last_undegraded":"2024-03-11T18:31:29.610504+0000","last_fullsized":"2024-03-11T18:31:29.610504+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:29.125099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584976+0000","last_change":"2024-03-11T18:30:32.311188+0000","last_active":"2024-03-11T18:31:29.584976+0000","last_peered":"2024-03-11T18:31:29.584976+0000","last_clean":"2024-03-11T18:31:29.584976+0000","last_became_active":"2024-03-11T18:30:32.310670+0000","last_became_peered":"2024-03-11T18:30:32.310670+0000","last_unstale":"2024-03-11T18:31:29.584976+0000","last_undegraded":"2024-03-11T18:31:29.584976+0000","last_fullsized":"2024-03-11T18:31:29.584976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:51:17.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.611088+0000","last_change":"2024-03-11T18:30:27.284990+0000","last_active":"2024-03-11T18:31:29.611088+0000","last_peered":"2024-03-11T18:31:29.611088+0000","last_clean":"2024-03-11T18:31:29.611088+0000","last_became_active":"2024-03-11T18:30:27.283629+0000","last_became_peered":"2024-03-11T18:30:27.283629+0000","last_unstale":"2024-03-11T18:31:29.611088+0000","last_undegraded":"2024-03-11T18:31:29.611088+0000","last_fullsized":"2024-03-11T18:31:29.611088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:33.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591401+0000","last_change":"2024-03-11T18:30:28.066345+0000","last_active":"2024-03-11T18:31:29.591401+0000","last_peered":"2024-03-11T18:31:29.591401+0000","last_clean":"2024-03-11T18:31:29.591401+0000","last_became_active":"2024-03-11T18:30:28.061654+0000","last_became_peered":"2024-03-11T18:30:28.061654+0000","last_unstale":"2024-03-11T18:31:29.591401+0000","last_undegraded":"2024-03-11T18:31:29.591401+0000","last_fullsized":"2024-03-11T18:31:29.591401+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:29.651510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585437+0000","last_change":"2024-03-11T18:30:32.318088+0000","last_active":"2024-03-11T18:31:29.585437+0000","last_peered":"2024-03-11T18:31:29.585437+0000","last_clean":"2024-03-11T18:31:29.585437+0000","last_became_active":"2024-03-11T18:30:32.317304+0000","last_became_peered":"2024-03-11T18:30:32.317304+0000","last_unstale":"2024-03-11T18:31:29.585437+0000","last_undegraded":"2024-03-11T18:31:29.585437+0000","last_fullsized":"2024-03-11T18:31:29.585437+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:01.678144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610319+0000","last_change":"2024-03-11T18:30:27.268370+0000","last_active":"2024-03-11T18:31:29.610319+0000","last_peered":"2024-03-11T18:31:29.610319+0000","last_clean":"2024-03-11T18:31:29.610319+0000","last_became_active":"2024-03-11T18:30:27.268033+0000","last_became_peered":"2024-03-11T18:30:27.268033+0000","last_unstale":"2024-03-11T18:31:29.610319+0000","last_undegraded":"2024-03-11T18:31:29.610319+0000","last_fullsized":"2024-03-11T18:31:29.610319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:31:13.545893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608975+0000","last_change":"2024-03-11T18:30:28.051611+0000","last_active":"2024-03-11T18:31:29.608975+0000","last_peered":"2024-03-11T18:31:29.608975+0000","last_clean":"2024-03-11T18:31:29.608975+0000","last_became_active":"2024-03-11T18:30:28.051249+0000","last_became_peered":"2024-03-11T18:30:28.051249+0000","last_unstale":"2024-03-11T18:31:29.608975+0000","last_undegraded":"2024-03-11T18:31:29.608975+0000","last_fullsized":"2024-03-11T18:31:29.608975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:12.942910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584639+0000","last_change":"2024-03-11T18:30:32.318934+0000","last_active":"2024-03-11T18:31:29.584639+0000","last_peered":"2024-03-11T18:31:29.584639+0000","last_clean":"2024-03-11T18:31:29.584639+0000","last_became_active":"2024-03-11T18:30:32.318474+0000","last_became_peered":"2024-03-11T18:30:32.318474+0000","last_unstale":"2024-03-11T18:31:29.584639+0000","last_undegraded":"2024-03-11T18:31:29.584639+0000","last_fullsized":"2024-03-11T18:31:29.584639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:48:33.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610073+0000","last_change":"2024-03-11T18:30:27.268675+0000","last_active":"2024-03-11T18:31:29.610073+0000","last_peered":"2024-03-11T18:31:29.610073+0000","last_clean":"2024-03-11T18:31:29.610073+0000","last_became_active":"2024-03-11T18:30:27.268320+0000","last_became_peered":"2024-03-11T18:30:27.268320+0000","last_unstale":"2024-03-11T18:31:29.610073+0000","last_undegraded":"2024-03-11T18:31:29.610073+0000","last_fullsized":"2024-03-11T18:31:29.610073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:06.439930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579354+0000","last_change":"2024-03-11T18:30:28.038380+0000","last_active":"2024-03-11T18:31:29.579354+0000","last_peered":"2024-03-11T18:31:29.579354+0000","last_clean":"2024-03-11T18:31:29.579354+0000","last_became_active":"2024-03-11T18:30:28.037878+0000","last_became_peered":"2024-03-11T18:30:28.037878+0000","last_unstale":"2024-03-11T18:31:29.579354+0000","last_undegraded":"2024-03-11T18:31:29.579354+0000","last_fullsized":"2024-03-11T18:31:29.579354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:10.988295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580594+0000","last_change":"2024-03-11T18:30:32.304747+0000","last_active":"2024-03-11T18:31:29.580594+0000","last_peered":"2024-03-11T18:31:29.580594+0000","last_clean":"2024-03-11T18:31:29.580594+0000","last_became_active":"2024-03-11T18:30:32.304282+0000","last_became_peered":"2024-03-11T18:30:32.304282+0000","last_unstale":"2024-03-11T18:31:29.580594+0000","last_undegraded":"2024-03-11T18:31:29.580594+0000","last_fullsized":"2024-03-11T18:31:29.580594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:42.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":128,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:19.667327+0000","last_change":"2024-03-11T18:31:29.590831+0000","last_active":"2024-03-11T18:32:19.667327+0000","last_peered":"2024-03-11T18:32:19.667327+0000","last_clean":"2024-03-11T18:32:19.667327+0000","last_became_active":"2024-03-11T18:31:29.586525+0000","last_became_peered":"2024-03-11T18:31:29.586525+0000","last_unstale":"2024-03-11T18:32:19.667327+0000","last_undegraded":"2024-03-11T18:32:19.667327+0000","last_fullsized":"2024-03-11T18:32:19.667327+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:04.712037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609917+0000","last_change":"2024-03-11T18:30:28.060280+0000","last_active":"2024-03-11T18:31:29.609917+0000","last_peered":"2024-03-11T18:31:29.609917+0000","last_clean":"2024-03-11T18:31:29.609917+0000","last_became_active":"2024-03-11T18:30:28.059929+0000","last_became_peered":"2024-03-11T18:30:28.059929+0000","last_unstale":"2024-03-11T18:31:29.609917+0000","last_undegraded":"2024-03-11T18:31:29.609917+0000","last_fullsized":"2024-03-11T18:31:29.609917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:47:31.995021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580852+0000","last_change":"2024-03-11T18:30:32.321754+0000","last_active":"2024-03-11T18:31:29.580852+0000","last_peered":"2024-03-11T18:31:29.580852+0000","last_clean":"2024-03-11T18:31:29.580852+0000","last_became_active":"2024-03-11T18:30:32.321329+0000","last_became_peered":"2024-03-11T18:30:32.321329+0000","last_unstale":"2024-03-11T18:31:29.580852+0000","last_undegraded":"2024-03-11T18:31:29.580852+0000","last_fullsized":"2024-03-11T18:31:29.580852+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:48.985005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609746+0000","last_change":"2024-03-11T18:30:27.285442+0000","last_active":"2024-03-11T18:31:29.609746+0000","last_peered":"2024-03-11T18:31:29.609746+0000","last_clean":"2024-03-11T18:31:29.609746+0000","last_became_active":"2024-03-11T18:30:27.285127+0000","last_became_peered":"2024-03-11T18:30:27.285127+0000","last_unstale":"2024-03-11T18:31:29.609746+0000","last_undegraded":"2024-03-11T18:31:29.609746+0000","last_fullsized":"2024-03-11T18:31:29.609746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:01.278846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.614546+0000","last_change":"2024-03-11T18:31:29.612985+0000","last_active":"2024-03-11T18:31:29.614546+0000","last_peered":"2024-03-11T18:31:29.614546+0000","last_clean":"2024-03-11T18:31:29.614546+0000","last_became_active":"2024-03-11T18:31:29.612340+0000","last_became_peered":"2024-03-11T18:31:29.612340+0000","last_unstale":"2024-03-11T18:31:29.614546+0000","last_undegraded":"2024-03-11T18:31:29.614546+0000","last_fullsized":"2024-03-11T18:31:29.614546+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:19:14.723344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586642+0000","last_change":"2024-03-11T18:30:32.327126+0000","last_active":"2024-03-11T18:31:29.586642+0000","last_peered":"2024-03-11T18:31:29.586642+0000","last_clean":"2024-03-11T18:31:29.586642+0000","last_became_active":"2024-03-11T18:30:32.326668+0000","last_became_peered":"2024-03-11T18:30:32.326668+0000","last_unstale":"2024-03-11T18:31:29.586642+0000","last_undegraded":"2024-03-11T18:31:29.586642+0000","last_fullsized":"2024-03-11T18:31:29.586642+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:03.231714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580792+0000","last_change":"2024-03-11T18:30:27.289439+0000","last_active":"2024-03-11T18:31:29.580792+0000","last_peered":"2024-03-11T18:31:29.580792+0000","last_clean":"2024-03-11T18:31:29.580792+0000","last_became_active":"2024-03-11T18:30:27.286067+0000","last_became_peered":"2024-03-11T18:30:27.286067+0000","last_unstale":"2024-03-11T18:31:29.580792+0000","last_undegraded":"2024-03-11T18:31:29.580792+0000","last_fullsized":"2024-03-11T18:31:29.580792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:35.522841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609641+0000","last_change":"2024-03-11T18:30:28.058525+0000","last_active":"2024-03-11T18:31:29.609641+0000","last_peered":"2024-03-11T18:31:29.609641+0000","last_clean":"2024-03-11T18:31:29.609641+0000","last_became_active":"2024-03-11T18:30:28.058122+0000","last_became_peered":"2024-03-11T18:30:28.058122+0000","last_unstale":"2024-03-11T18:31:29.609641+0000","last_undegraded":"2024-03-11T18:31:29.609641+0000","last_fullsized":"2024-03-11T18:31:29.609641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:45:04.537880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591536+0000","last_change":"2024-03-11T18:30:32.314757+0000","last_active":"2024-03-11T18:31:29.591536+0000","last_peered":"2024-03-11T18:31:29.591536+0000","last_clean":"2024-03-11T18:31:29.591536+0000","last_became_active":"2024-03-11T18:30:32.313461+0000","last_became_peered":"2024-03-11T18:30:32.313461+0000","last_unstale":"2024-03-11T18:31:29.591536+0000","last_undegraded":"2024-03-11T18:31:29.591536+0000","last_fullsized":"2024-03-11T18:31:29.591536+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:11:27.881788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578835+0000","last_change":"2024-03-11T18:30:27.268359+0000","last_active":"2024-03-11T18:31:29.578835+0000","last_peered":"2024-03-11T18:31:29.578835+0000","last_clean":"2024-03-11T18:31:29.578835+0000","last_became_active":"2024-03-11T18:30:27.268045+0000","last_became_peered":"2024-03-11T18:30:27.268045+0000","last_unstale":"2024-03-11T18:31:29.578835+0000","last_undegraded":"2024-03-11T18:31:29.578835+0000","last_fullsized":"2024-03-11T18:31:29.578835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:53.678477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590413+0000","last_change":"2024-03-11T18:30:28.051331+0000","last_active":"2024-03-11T18:31:29.590413+0000","last_peered":"2024-03-11T18:31:29.590413+0000","last_clean":"2024-03-11T18:31:29.590413+0000","last_became_active":"2024-03-11T18:30:28.050462+0000","last_became_peered":"2024-03-11T18:30:28.050462+0000","last_unstale":"2024-03-11T18:31:29.590413+0000","last_undegraded":"2024-03-11T18:31:29.590413+0000","last_fullsized":"2024-03-11T18:31:29.590413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:24.194650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609940+0000","last_change":"2024-03-11T18:30:32.317173+0000","last_active":"2024-03-11T18:31:29.609940+0000","last_peered":"2024-03-11T18:31:29.609940+0000","last_clean":"2024-03-11T18:31:29.609940+0000","last_became_active":"2024-03-11T18:30:32.316721+0000","last_became_peered":"2024-03-11T18:30:32.316721+0000","last_unstale":"2024-03-11T18:31:29.609940+0000","last_undegraded":"2024-03-11T18:31:29.609940+0000","last_fullsized":"2024-03-11T18:31:29.609940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:43.665949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:17.663599+0000","last_change":"2024-03-11T18:30:27.279586+0000","last_active":"2024-03-11T18:32:17.663599+0000","last_peered":"2024-03-11T18:32:17.663599+0000","last_clean":"2024-03-11T18:32:17.663599+0000","last_became_active":"2024-03-11T18:30:27.279262+0000","last_became_peered":"2024-03-11T18:30:27.279262+0000","last_unstale":"2024-03-11T18:32:17.663599+0000","last_undegraded":"2024-03-11T18:32:17.663599+0000","last_fullsized":"2024-03-11T18:32:17.663599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:02:33.570755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609286+0000","last_change":"2024-03-11T18:30:28.060636+0000","last_active":"2024-03-11T18:31:29.609286+0000","last_peered":"2024-03-11T18:31:29.609286+0000","last_clean":"2024-03-11T18:31:29.609286+0000","last_became_active":"2024-03-11T18:30:28.060282+0000","last_became_peered":"2024-03-11T18:30:28.060282+0000","last_unstale":"2024-03-11T18:31:29.609286+0000","last_undegraded":"2024-03-11T18:31:29.609286+0000","last_fullsized":"2024-03-11T18:31:29.609286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:56.618563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.589505+0000","last_change":"2024-03-11T18:30:32.314680+0000","last_active":"2024-03-11T18:31:29.589505+0000","last_peered":"2024-03-11T18:31:29.589505+0000","last_clean":"2024-03-11T18:31:29.589505+0000","last_became_active":"2024-03-11T18:30:32.314102+0000","last_became_peered":"2024-03-11T18:30:32.314102+0000","last_unstale":"2024-03-11T18:31:29.589505+0000","last_undegraded":"2024-03-11T18:31:29.589505+0000","last_fullsized":"2024-03-11T18:31:29.589505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:18:55.989746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584409+0000","last_change":"2024-03-11T18:30:27.267028+0000","last_active":"2024-03-11T18:31:29.584409+0000","last_peered":"2024-03-11T18:31:29.584409+0000","last_clean":"2024-03-11T18:31:29.584409+0000","last_became_active":"2024-03-11T18:30:27.266712+0000","last_became_peered":"2024-03-11T18:30:27.266712+0000","last_unstale":"2024-03-11T18:31:29.584409+0000","last_undegraded":"2024-03-11T18:31:29.584409+0000","last_fullsized":"2024-03-11T18:31:29.584409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:35.978113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591112+0000","last_change":"2024-03-11T18:30:28.066607+0000","last_active":"2024-03-11T18:31:29.591112+0000","last_peered":"2024-03-11T18:31:29.591112+0000","last_clean":"2024-03-11T18:31:29.591112+0000","last_became_active":"2024-03-11T18:30:28.061181+0000","last_became_peered":"2024-03-11T18:30:28.061181+0000","last_unstale":"2024-03-11T18:31:29.591112+0000","last_undegraded":"2024-03-11T18:31:29.591112+0000","last_fullsized":"2024-03-11T18:31:29.591112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:01.203632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.612447+0000","last_change":"2024-03-11T18:31:29.612447+0000","last_active":"2024-03-11T18:31:29.612447+0000","last_peered":"2024-03-11T18:31:29.612447+0000","last_clean":"2024-03-11T18:31:29.612447+0000","last_became_active":"2024-03-11T18:31:29.611143+0000","last_became_peered":"2024-03-11T18:31:29.611143+0000","last_unstale":"2024-03-11T18:31:29.612447+0000","last_undegraded":"2024-03-11T18:31:29.612447+0000","last_fullsized":"2024-03-11T18:31:29.612447+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:23.662676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579244+0000","last_change":"2024-03-11T18:30:27.270147+0000","last_active":"2024-03-11T18:31:29.579244+0000","last_peered":"2024-03-11T18:31:29.579244+0000","last_clean":"2024-03-11T18:31:29.579244+0000","last_became_active":"2024-03-11T18:30:27.269791+0000","last_became_peered":"2024-03-11T18:30:27.269791+0000","last_unstale":"2024-03-11T18:31:29.579244+0000","last_undegraded":"2024-03-11T18:31:29.579244+0000","last_fullsized":"2024-03-11T18:31:29.579244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:22.007518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609088+0000","last_change":"2024-03-11T18:30:28.035357+0000","last_active":"2024-03-11T18:31:29.609088+0000","last_peered":"2024-03-11T18:31:29.609088+0000","last_clean":"2024-03-11T18:31:29.609088+0000","last_became_active":"2024-03-11T18:30:28.035005+0000","last_became_peered":"2024-03-11T18:30:28.035005+0000","last_unstale":"2024-03-11T18:31:29.609088+0000","last_undegraded":"2024-03-11T18:31:29.609088+0000","last_fullsized":"2024-03-11T18:31:29.609088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:21.450440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.589644+0000","last_change":"2024-03-11T18:30:32.323472+0000","last_active":"2024-03-11T18:31:29.589644+0000","last_peered":"2024-03-11T18:31:29.589644+0000","last_clean":"2024-03-11T18:31:29.589644+0000","last_became_active":"2024-03-11T18:30:32.321590+0000","last_became_peered":"2024-03-11T18:30:32.321590+0000","last_unstale":"2024-03-11T18:31:29.589644+0000","last_undegraded":"2024-03-11T18:31:29.589644+0000","last_fullsized":"2024-03-11T18:31:29.589644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:12.329127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'33","reported_seq":223,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:18.665091+0000","last_change":"2024-03-11T18:30:27.267683+0000","last_active":"2024-03-11T18:32:18.665091+0000","last_peered":"2024-03-11T18:32:18.665091+0000","last_clean":"2024-03-11T18:32:18.665091+0000","last_became_active":"2024-03-11T18:30:27.267333+0000","last_became_peered":"2024-03-11T18:30:27.267333+0000","last_unstale":"2024-03-11T18:32:18.665091+0000","last_undegraded":"2024-03-11T18:32:18.665091+0000","last_fullsized":"2024-03-11T18:32:18.665091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:33:18.721843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":164,"num_read_kb":144,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610140+0000","last_change":"2024-03-11T18:30:28.038846+0000","last_active":"2024-03-11T18:31:29.610140+0000","last_peered":"2024-03-11T18:31:29.610140+0000","last_clean":"2024-03-11T18:31:29.610140+0000","last_became_active":"2024-03-11T18:30:28.038479+0000","last_became_peered":"2024-03-11T18:30:28.038479+0000","last_unstale":"2024-03-11T18:31:29.610140+0000","last_undegraded":"2024-03-11T18:31:29.610140+0000","last_fullsized":"2024-03-11T18:31:29.610140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:25.877627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579658+0000","last_change":"2024-03-11T18:30:32.317761+0000","last_active":"2024-03-11T18:31:29.579658+0000","last_peered":"2024-03-11T18:31:29.579658+0000","last_clean":"2024-03-11T18:31:29.579658+0000","last_became_active":"2024-03-11T18:30:32.317438+0000","last_became_peered":"2024-03-11T18:30:32.317438+0000","last_unstale":"2024-03-11T18:31:29.579658+0000","last_undegraded":"2024-03-11T18:31:29.579658+0000","last_fullsized":"2024-03-11T18:31:29.579658+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:58.145037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581109+0000","last_change":"2024-03-11T18:30:27.276641+0000","last_active":"2024-03-11T18:31:29.581109+0000","last_peered":"2024-03-11T18:31:29.581109+0000","last_clean":"2024-03-11T18:31:29.581109+0000","last_became_active":"2024-03-11T18:30:27.276293+0000","last_became_peered":"2024-03-11T18:30:27.276293+0000","last_unstale":"2024-03-11T18:31:29.581109+0000","last_undegraded":"2024-03-11T18:31:29.581109+0000","last_fullsized":"2024-03-11T18:31:29.581109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:46.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580612+0000","last_change":"2024-03-11T18:30:28.064304+0000","last_active":"2024-03-11T18:31:29.580612+0000","last_peered":"2024-03-11T18:31:29.580612+0000","last_clean":"2024-03-11T18:31:29.580612+0000","last_became_active":"2024-03-11T18:30:28.063923+0000","last_became_peered":"2024-03-11T18:30:28.063923+0000","last_unstale":"2024-03-11T18:31:29.580612+0000","last_undegraded":"2024-03-11T18:31:29.580612+0000","last_fullsized":"2024-03-11T18:31:29.580612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:57:41.289717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586465+0000","last_change":"2024-03-11T18:30:32.325409+0000","last_active":"2024-03-11T18:31:29.586465+0000","last_peered":"2024-03-11T18:31:29.586465+0000","last_clean":"2024-03-11T18:31:29.586465+0000","last_became_active":"2024-03-11T18:30:32.324953+0000","last_became_peered":"2024-03-11T18:30:32.324953+0000","last_unstale":"2024-03-11T18:31:29.586465+0000","last_undegraded":"2024-03-11T18:31:29.586465+0000","last_fullsized":"2024-03-11T18:31:29.586465+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:44:54.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579334+0000","last_change":"2024-03-11T18:30:27.271690+0000","last_active":"2024-03-11T18:31:29.579334+0000","last_peered":"2024-03-11T18:31:29.579334+0000","last_clean":"2024-03-11T18:31:29.579334+0000","last_became_active":"2024-03-11T18:30:27.271429+0000","last_became_peered":"2024-03-11T18:30:27.271429+0000","last_unstale":"2024-03-11T18:31:29.579334+0000","last_undegraded":"2024-03-11T18:31:29.579334+0000","last_fullsized":"2024-03-11T18:31:29.579334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:55:23.896642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583790+0000","last_change":"2024-03-11T18:30:28.054214+0000","last_active":"2024-03-11T18:31:29.583790+0000","last_peered":"2024-03-11T18:31:29.583790+0000","last_clean":"2024-03-11T18:31:29.583790+0000","last_became_active":"2024-03-11T18:30:28.053447+0000","last_became_peered":"2024-03-11T18:30:28.053447+0000","last_unstale":"2024-03-11T18:31:29.583790+0000","last_undegraded":"2024-03-11T18:31:29.583790+0000","last_fullsized":"2024-03-11T18:31:29.583790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:08:49.866509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594999+0000","last_change":"2024-03-11T18:30:32.325028+0000","last_active":"2024-03-11T18:31:29.594999+0000","last_peered":"2024-03-11T18:31:29.594999+0000","last_clean":"2024-03-11T18:31:29.594999+0000","last_became_active":"2024-03-11T18:30:32.324526+0000","last_became_peered":"2024-03-11T18:30:32.324526+0000","last_unstale":"2024-03-11T18:31:29.594999+0000","last_undegraded":"2024-03-11T18:31:29.594999+0000","last_fullsized":"2024-03-11T18:31:29.594999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:08.031224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:20.620942+0000","last_change":"2024-03-11T18:30:27.265284+0000","last_active":"2024-03-11T18:32:20.620942+0000","last_peered":"2024-03-11T18:32:20.620942+0000","last_clean":"2024-03-11T18:32:20.620942+0000","last_became_active":"2024-03-11T18:30:27.264402+0000","last_became_peered":"2024-03-11T18:30:27.264402+0000","last_unstale":"2024-03-11T18:32:20.620942+0000","last_undegraded":"2024-03-11T18:32:20.620942+0000","last_fullsized":"2024-03-11T18:32:20.620942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:14.768956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583967+0000","last_change":"2024-03-11T18:30:28.040540+0000","last_active":"2024-03-11T18:31:29.583967+0000","last_peered":"2024-03-11T18:31:29.583967+0000","last_clean":"2024-03-11T18:31:29.583967+0000","last_became_active":"2024-03-11T18:30:28.040181+0000","last_became_peered":"2024-03-11T18:30:28.040181+0000","last_unstale":"2024-03-11T18:31:29.583967+0000","last_undegraded":"2024-03-11T18:31:29.583967+0000","last_fullsized":"2024-03-11T18:31:29.583967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:19.079250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.582353+0000","last_change":"2024-03-11T18:30:32.326182+0000","last_active":"2024-03-11T18:31:29.582353+0000","last_peered":"2024-03-11T18:31:29.582353+0000","last_clean":"2024-03-11T18:31:29.582353+0000","last_became_active":"2024-03-11T18:30:32.325793+0000","last_became_peered":"2024-03-11T18:30:32.325793+0000","last_unstale":"2024-03-11T18:31:29.582353+0000","last_undegraded":"2024-03-11T18:31:29.582353+0000","last_fullsized":"2024-03-11T18:31:29.582353+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:14:13.212511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609646+0000","last_change":"2024-03-11T18:30:27.266724+0000","last_active":"2024-03-11T18:31:29.609646+0000","last_peered":"2024-03-11T18:31:29.609646+0000","last_clean":"2024-03-11T18:31:29.609646+0000","last_became_active":"2024-03-11T18:30:27.266394+0000","last_became_peered":"2024-03-11T18:30:27.266394+0000","last_unstale":"2024-03-11T18:31:29.609646+0000","last_undegraded":"2024-03-11T18:31:29.609646+0000","last_fullsized":"2024-03-11T18:31:29.609646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:24.115722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579253+0000","last_change":"2024-03-11T18:30:28.039476+0000","last_active":"2024-03-11T18:31:29.579253+0000","last_peered":"2024-03-11T18:31:29.579253+0000","last_clean":"2024-03-11T18:31:29.579253+0000","last_became_active":"2024-03-11T18:30:28.038897+0000","last_became_peered":"2024-03-11T18:30:28.038897+0000","last_unstale":"2024-03-11T18:31:29.579253+0000","last_undegraded":"2024-03-11T18:31:29.579253+0000","last_fullsized":"2024-03-11T18:31:29.579253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:36:16.103173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578819+0000","last_change":"2024-03-11T18:30:32.319516+0000","last_active":"2024-03-11T18:31:29.578819+0000","last_peered":"2024-03-11T18:31:29.578819+0000","last_clean":"2024-03-11T18:31:29.578819+0000","last_became_active":"2024-03-11T18:30:32.319117+0000","last_became_peered":"2024-03-11T18:30:32.319117+0000","last_unstale":"2024-03-11T18:31:29.578819+0000","last_undegraded":"2024-03-11T18:31:29.578819+0000","last_fullsized":"2024-03-11T18:31:29.578819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:39.797872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610559+0000","last_change":"2024-03-11T18:30:27.267543+0000","last_active":"2024-03-11T18:31:29.610559+0000","last_peered":"2024-03-11T18:31:29.610559+0000","last_clean":"2024-03-11T18:31:29.610559+0000","last_became_active":"2024-03-11T18:30:27.267235+0000","last_became_peered":"2024-03-11T18:30:27.267235+0000","last_unstale":"2024-03-11T18:31:29.610559+0000","last_undegraded":"2024-03-11T18:31:29.610559+0000","last_fullsized":"2024-03-11T18:31:29.610559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:24.556896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586578+0000","last_change":"2024-03-11T18:30:28.039401+0000","last_active":"2024-03-11T18:31:29.586578+0000","last_peered":"2024-03-11T18:31:29.586578+0000","last_clean":"2024-03-11T18:31:29.586578+0000","last_became_active":"2024-03-11T18:30:28.039054+0000","last_became_peered":"2024-03-11T18:30:28.039054+0000","last_unstale":"2024-03-11T18:31:29.586578+0000","last_undegraded":"2024-03-11T18:31:29.586578+0000","last_fullsized":"2024-03-11T18:31:29.586578+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:01.701304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609860+0000","last_change":"2024-03-11T18:30:32.527930+0000","last_active":"2024-03-11T18:31:29.609860+0000","last_peered":"2024-03-11T18:31:29.609860+0000","last_clean":"2024-03-11T18:31:29.609860+0000","last_became_active":"2024-03-11T18:30:32.527371+0000","last_became_peered":"2024-03-11T18:30:32.527371+0000","last_unstale":"2024-03-11T18:31:29.609860+0000","last_undegraded":"2024-03-11T18:31:29.609860+0000","last_fullsized":"2024-03-11T18:31:29.609860+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:09:52.361924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591575+0000","last_change":"2024-03-11T18:30:27.281098+0000","last_active":"2024-03-11T18:31:29.591575+0000","last_peered":"2024-03-11T18:31:29.591575+0000","last_clean":"2024-03-11T18:31:29.591575+0000","last_became_active":"2024-03-11T18:30:27.280796+0000","last_became_peered":"2024-03-11T18:30:27.280796+0000","last_unstale":"2024-03-11T18:31:29.591575+0000","last_undegraded":"2024-03-11T18:31:29.591575+0000","last_fullsized":"2024-03-11T18:31:29.591575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:28.661626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609825+0000","last_change":"2024-03-11T18:30:28.054079+0000","last_active":"2024-03-11T18:31:29.609825+0000","last_peered":"2024-03-11T18:31:29.609825+0000","last_clean":"2024-03-11T18:31:29.609825+0000","last_became_active":"2024-03-11T18:30:28.053715+0000","last_became_peered":"2024-03-11T18:30:28.053715+0000","last_unstale":"2024-03-11T18:31:29.609825+0000","last_undegraded":"2024-03-11T18:31:29.609825+0000","last_fullsized":"2024-03-11T18:31:29.609825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:05.211602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581769+0000","last_change":"2024-03-11T18:30:32.311132+0000","last_active":"2024-03-11T18:31:29.581769+0000","last_peered":"2024-03-11T18:31:29.581769+0000","last_clean":"2024-03-11T18:31:29.581769+0000","last_became_active":"2024-03-11T18:30:32.309585+0000","last_became_peered":"2024-03-11T18:30:32.309585+0000","last_unstale":"2024-03-11T18:31:29.581769+0000","last_undegraded":"2024-03-11T18:31:29.581769+0000","last_fullsized":"2024-03-11T18:31:29.581769+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:57.272479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584220+0000","last_change":"2024-03-11T18:30:27.288080+0000","last_active":"2024-03-11T18:31:29.584220+0000","last_peered":"2024-03-11T18:31:29.584220+0000","last_clean":"2024-03-11T18:31:29.584220+0000","last_became_active":"2024-03-11T18:30:27.287776+0000","last_became_peered":"2024-03-11T18:30:27.287776+0000","last_unstale":"2024-03-11T18:31:29.584220+0000","last_undegraded":"2024-03-11T18:31:29.584220+0000","last_fullsized":"2024-03-11T18:31:29.584220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:22.081753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609889+0000","last_change":"2024-03-11T18:30:28.062011+0000","last_active":"2024-03-11T18:31:29.609889+0000","last_peered":"2024-03-11T18:31:29.609889+0000","last_clean":"2024-03-11T18:31:29.609889+0000","last_became_active":"2024-03-11T18:30:28.061592+0000","last_became_peered":"2024-03-11T18:30:28.061592+0000","last_unstale":"2024-03-11T18:31:29.609889+0000","last_undegraded":"2024-03-11T18:31:29.609889+0000","last_fullsized":"2024-03-11T18:31:29.609889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:10.013980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581357+0000","last_change":"2024-03-11T18:30:32.528123+0000","last_active":"2024-03-11T18:31:29.581357+0000","last_peered":"2024-03-11T18:31:29.581357+0000","last_clean":"2024-03-11T18:31:29.581357+0000","last_became_active":"2024-03-11T18:30:32.527611+0000","last_became_peered":"2024-03-11T18:30:32.527611+0000","last_unstale":"2024-03-11T18:31:29.581357+0000","last_undegraded":"2024-03-11T18:31:29.581357+0000","last_fullsized":"2024-03-11T18:31:29.581357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:28.501667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578964+0000","last_change":"2024-03-11T18:30:28.036930+0000","last_active":"2024-03-11T18:31:29.578964+0000","last_peered":"2024-03-11T18:31:29.578964+0000","last_clean":"2024-03-11T18:31:29.578964+0000","last_became_active":"2024-03-11T18:30:28.036286+0000","last_became_peered":"2024-03-11T18:30:28.036286+0000","last_unstale":"2024-03-11T18:31:29.578964+0000","last_undegraded":"2024-03-11T18:31:29.578964+0000","last_fullsized":"2024-03-11T18:31:29.578964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:10:40.239284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":111,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:19.618639+0000","last_change":"2024-03-11T18:30:27.287838+0000","last_active":"2024-03-11T18:32:19.618639+0000","last_peered":"2024-03-11T18:32:19.618639+0000","last_clean":"2024-03-11T18:32:19.618639+0000","last_became_active":"2024-03-11T18:30:27.287447+0000","last_became_peered":"2024-03-11T18:30:27.287447+0000","last_unstale":"2024-03-11T18:32:19.618639+0000","last_undegraded":"2024-03-11T18:32:19.618639+0000","last_fullsized":"2024-03-11T18:32:19.618639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:47.242977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.617874+0000","last_change":"2024-03-11T18:31:29.617874+0000","last_active":"2024-03-11T18:31:29.617874+0000","last_peered":"2024-03-11T18:31:29.617874+0000","last_clean":"2024-03-11T18:31:29.617874+0000","last_became_active":"2024-03-11T18:31:29.610902+0000","last_became_peered":"2024-03-11T18:31:29.610902+0000","last_unstale":"2024-03-11T18:31:29.617874+0000","last_undegraded":"2024-03-11T18:31:29.617874+0000","last_fullsized":"2024-03-11T18:31:29.617874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:29:23.078025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610215+0000","last_change":"2024-03-11T18:30:28.037906+0000","last_active":"2024-03-11T18:31:29.610215+0000","last_peered":"2024-03-11T18:31:29.610215+0000","last_clean":"2024-03-11T18:31:29.610215+0000","last_became_active":"2024-03-11T18:30:28.034143+0000","last_became_peered":"2024-03-11T18:30:28.034143+0000","last_unstale":"2024-03-11T18:31:29.610215+0000","last_undegraded":"2024-03-11T18:31:29.610215+0000","last_fullsized":"2024-03-11T18:31:29.610215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:36.728143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585543+0000","last_change":"2024-03-11T18:30:27.273520+0000","last_active":"2024-03-11T18:31:29.585543+0000","last_peered":"2024-03-11T18:31:29.585543+0000","last_clean":"2024-03-11T18:31:29.585543+0000","last_became_active":"2024-03-11T18:30:27.273141+0000","last_became_peered":"2024-03-11T18:30:27.273141+0000","last_unstale":"2024-03-11T18:31:29.585543+0000","last_undegraded":"2024-03-11T18:31:29.585543+0000","last_fullsized":"2024-03-11T18:31:29.585543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:51.467579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":362,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583899+0000","last_change":"2024-03-11T18:29:08.539862+0000","last_active":"2024-03-11T18:31:29.583899+0000","last_peered":"2024-03-11T18:31:29.583899+0000","last_clean":"2024-03-11T18:31:29.583899+0000","last_became_active":"2024-03-11T18:29:08.539403+0000","last_became_peered":"2024-03-11T18:29:08.539403+0000","last_unstale":"2024-03-11T18:31:29.583899+0000","last_undegraded":"2024-03-11T18:31:29.583899+0000","last_fullsized":"2024-03-11T18:31:29.583899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_clean_scrub_stamp":"2024-03-11T18:25:13.035017+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:24.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579774+0000","last_change":"2024-03-11T18:30:32.528841+0000","last_active":"2024-03-11T18:31:29.579774+0000","last_peered":"2024-03-11T18:31:29.579774+0000","last_clean":"2024-03-11T18:31:29.579774+0000","last_became_active":"2024-03-11T18:30:32.528320+0000","last_became_peered":"2024-03-11T18:30:32.528320+0000","last_unstale":"2024-03-11T18:31:29.579774+0000","last_undegraded":"2024-03-11T18:31:29.579774+0000","last_fullsized":"2024-03-11T18:31:29.579774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:26.354490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580410+0000","last_change":"2024-03-11T18:30:28.047552+0000","last_active":"2024-03-11T18:31:29.580410+0000","last_peered":"2024-03-11T18:31:29.580410+0000","last_clean":"2024-03-11T18:31:29.580410+0000","last_became_active":"2024-03-11T18:30:28.046470+0000","last_became_peered":"2024-03-11T18:30:28.046470+0000","last_unstale":"2024-03-11T18:31:29.580410+0000","last_undegraded":"2024-03-11T18:31:29.580410+0000","last_fullsized":"2024-03-11T18:31:29.580410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:58:23.080406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584085+0000","last_change":"2024-03-11T18:30:27.269416+0000","last_active":"2024-03-11T18:31:29.584085+0000","last_peered":"2024-03-11T18:31:29.584085+0000","last_clean":"2024-03-11T18:31:29.584085+0000","last_became_active":"2024-03-11T18:30:27.268951+0000","last_became_peered":"2024-03-11T18:30:27.268951+0000","last_unstale":"2024-03-11T18:31:29.584085+0000","last_undegraded":"2024-03-11T18:31:29.584085+0000","last_fullsized":"2024-03-11T18:31:29.584085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:03:16.138585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586398+0000","last_change":"2024-03-11T18:30:32.309247+0000","last_active":"2024-03-11T18:31:29.586398+0000","last_peered":"2024-03-11T18:31:29.586398+0000","last_clean":"2024-03-11T18:31:29.586398+0000","last_became_active":"2024-03-11T18:30:32.308761+0000","last_became_peered":"2024-03-11T18:30:32.308761+0000","last_unstale":"2024-03-11T18:31:29.586398+0000","last_undegraded":"2024-03-11T18:31:29.586398+0000","last_fullsized":"2024-03-11T18:31:29.586398+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:10.525154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610171+0000","last_change":"2024-03-11T18:30:27.279543+0000","last_active":"2024-03-11T18:31:29.610171+0000","last_peered":"2024-03-11T18:31:29.610171+0000","last_clean":"2024-03-11T18:31:29.610171+0000","last_became_active":"2024-03-11T18:30:27.279203+0000","last_became_peered":"2024-03-11T18:30:27.279203+0000","last_unstale":"2024-03-11T18:31:29.610171+0000","last_undegraded":"2024-03-11T18:31:29.610171+0000","last_fullsized":"2024-03-11T18:31:29.610171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:31.792757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581955+0000","last_change":"2024-03-11T18:30:28.060508+0000","last_active":"2024-03-11T18:31:29.581955+0000","last_peered":"2024-03-11T18:31:29.581955+0000","last_clean":"2024-03-11T18:31:29.581955+0000","last_became_active":"2024-03-11T18:30:28.059074+0000","last_became_peered":"2024-03-11T18:30:28.059074+0000","last_unstale":"2024-03-11T18:31:29.581955+0000","last_undegraded":"2024-03-11T18:31:29.581955+0000","last_fullsized":"2024-03-11T18:31:29.581955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:34.204704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609508+0000","last_change":"2024-03-11T18:30:32.528017+0000","last_active":"2024-03-11T18:31:29.609508+0000","last_peered":"2024-03-11T18:31:29.609508+0000","last_clean":"2024-03-11T18:31:29.609508+0000","last_became_active":"2024-03-11T18:30:32.527518+0000","last_became_peered":"2024-03-11T18:30:32.527518+0000","last_unstale":"2024-03-11T18:31:29.609508+0000","last_undegraded":"2024-03-11T18:31:29.609508+0000","last_fullsized":"2024-03-11T18:31:29.609508+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:13:09.095889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594176+0000","last_change":"2024-03-11T18:30:27.275540+0000","last_active":"2024-03-11T18:31:29.594176+0000","last_peered":"2024-03-11T18:31:29.594176+0000","last_clean":"2024-03-11T18:31:29.594176+0000","last_became_active":"2024-03-11T18:30:27.275070+0000","last_became_peered":"2024-03-11T18:30:27.275070+0000","last_unstale":"2024-03-11T18:31:29.594176+0000","last_undegraded":"2024-03-11T18:31:29.594176+0000","last_fullsized":"2024-03-11T18:31:29.594176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:59.729542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610628+0000","last_change":"2024-03-11T18:30:28.045096+0000","last_active":"2024-03-11T18:31:29.610628+0000","last_peered":"2024-03-11T18:31:29.610628+0000","last_clean":"2024-03-11T18:31:29.610628+0000","last_became_active":"2024-03-11T18:30:28.044408+0000","last_became_peered":"2024-03-11T18:30:28.044408+0000","last_unstale":"2024-03-11T18:31:29.610628+0000","last_undegraded":"2024-03-11T18:31:29.610628+0000","last_fullsized":"2024-03-11T18:31:29.610628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:49:59.426247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580539+0000","last_change":"2024-03-11T18:30:32.303288+0000","last_active":"2024-03-11T18:31:29.580539+0000","last_peered":"2024-03-11T18:31:29.580539+0000","last_clean":"2024-03-11T18:31:29.580539+0000","last_became_active":"2024-03-11T18:30:32.302746+0000","last_became_peered":"2024-03-11T18:30:32.302746+0000","last_unstale":"2024-03-11T18:31:29.580539+0000","last_undegraded":"2024-03-11T18:31:29.580539+0000","last_fullsized":"2024-03-11T18:31:29.580539+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:24.541448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584867+0000","last_change":"2024-03-11T18:30:27.272880+0000","last_active":"2024-03-11T18:31:29.584867+0000","last_peered":"2024-03-11T18:31:29.584867+0000","last_clean":"2024-03-11T18:31:29.584867+0000","last_became_active":"2024-03-11T18:30:27.272611+0000","last_became_peered":"2024-03-11T18:30:27.272611+0000","last_unstale":"2024-03-11T18:31:29.584867+0000","last_undegraded":"2024-03-11T18:31:29.584867+0000","last_fullsized":"2024-03-11T18:31:29.584867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:41:10.322990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591866+0000","last_change":"2024-03-11T18:30:28.065613+0000","last_active":"2024-03-11T18:31:29.591866+0000","last_peered":"2024-03-11T18:31:29.591866+0000","last_clean":"2024-03-11T18:31:29.591866+0000","last_became_active":"2024-03-11T18:30:28.060520+0000","last_became_peered":"2024-03-11T18:30:28.060520+0000","last_unstale":"2024-03-11T18:31:29.591866+0000","last_undegraded":"2024-03-11T18:31:29.591866+0000","last_fullsized":"2024-03-11T18:31:29.591866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:44.530168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.611177+0000","last_change":"2024-03-11T18:31:29.611177+0000","last_active":"2024-03-11T18:31:29.611177+0000","last_peered":"2024-03-11T18:31:29.611177+0000","last_clean":"2024-03-11T18:31:29.611177+0000","last_became_active":"2024-03-11T18:31:29.610717+0000","last_became_peered":"2024-03-11T18:31:29.610717+0000","last_unstale":"2024-03-11T18:31:29.611177+0000","last_undegraded":"2024-03-11T18:31:29.611177+0000","last_fullsized":"2024-03-11T18:31:29.611177+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:41.226064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,8],"acting":[4,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608868+0000","last_change":"2024-03-11T18:30:27.261435+0000","last_active":"2024-03-11T18:31:29.608868+0000","last_peered":"2024-03-11T18:31:29.608868+0000","last_clean":"2024-03-11T18:31:29.608868+0000","last_became_active":"2024-03-11T18:30:27.260070+0000","last_became_peered":"2024-03-11T18:30:27.260070+0000","last_unstale":"2024-03-11T18:31:29.608868+0000","last_undegraded":"2024-03-11T18:31:29.608868+0000","last_fullsized":"2024-03-11T18:31:29.608868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:11.785818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609774+0000","last_change":"2024-03-11T18:30:28.041708+0000","last_active":"2024-03-11T18:31:29.609774+0000","last_peered":"2024-03-11T18:31:29.609774+0000","last_clean":"2024-03-11T18:31:29.609774+0000","last_became_active":"2024-03-11T18:30:28.039647+0000","last_became_peered":"2024-03-11T18:30:28.039647+0000","last_unstale":"2024-03-11T18:31:29.609774+0000","last_undegraded":"2024-03-11T18:31:29.609774+0000","last_fullsized":"2024-03-11T18:31:29.609774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.624175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.596205+0000","last_change":"2024-03-11T18:30:32.326512+0000","last_active":"2024-03-11T18:31:29.596205+0000","last_peered":"2024-03-11T18:31:29.596205+0000","last_clean":"2024-03-11T18:31:29.596205+0000","last_became_active":"2024-03-11T18:30:32.325253+0000","last_became_peered":"2024-03-11T18:30:32.325253+0000","last_unstale":"2024-03-11T18:31:29.596205+0000","last_undegraded":"2024-03-11T18:31:29.596205+0000","last_fullsized":"2024-03-11T18:31:29.596205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:48.589769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609610+0000","last_change":"2024-03-11T18:30:27.286194+0000","last_active":"2024-03-11T18:31:29.609610+0000","last_peered":"2024-03-11T18:31:29.609610+0000","last_clean":"2024-03-11T18:31:29.609610+0000","last_became_active":"2024-03-11T18:30:27.284877+0000","last_became_peered":"2024-03-11T18:30:27.284877+0000","last_unstale":"2024-03-11T18:31:29.609610+0000","last_undegraded":"2024-03-11T18:31:29.609610+0000","last_fullsized":"2024-03-11T18:31:29.609610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:56:20.324266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586216+0000","last_change":"2024-03-11T18:30:28.061038+0000","last_active":"2024-03-11T18:31:29.586216+0000","last_peered":"2024-03-11T18:31:29.586216+0000","last_clean":"2024-03-11T18:31:29.586216+0000","last_became_active":"2024-03-11T18:30:28.060701+0000","last_became_peered":"2024-03-11T18:30:28.060701+0000","last_unstale":"2024-03-11T18:31:29.586216+0000","last_undegraded":"2024-03-11T18:31:29.586216+0000","last_fullsized":"2024-03-11T18:31:29.586216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:11.838309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585027+0000","last_change":"2024-03-11T18:30:32.316762+0000","last_active":"2024-03-11T18:31:29.585027+0000","last_peered":"2024-03-11T18:31:29.585027+0000","last_clean":"2024-03-11T18:31:29.585027+0000","last_became_active":"2024-03-11T18:30:32.316318+0000","last_became_peered":"2024-03-11T18:30:32.316318+0000","last_unstale":"2024-03-11T18:31:29.585027+0000","last_undegraded":"2024-03-11T18:31:29.585027+0000","last_fullsized":"2024-03-11T18:31:29.585027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:29.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580546+0000","last_change":"2024-03-11T18:30:27.276636+0000","last_active":"2024-03-11T18:31:29.580546+0000","last_peered":"2024-03-11T18:31:29.580546+0000","last_clean":"2024-03-11T18:31:29.580546+0000","last_became_active":"2024-03-11T18:30:27.276290+0000","last_became_peered":"2024-03-11T18:30:27.276290+0000","last_unstale":"2024-03-11T18:31:29.580546+0000","last_undegraded":"2024-03-11T18:31:29.580546+0000","last_fullsized":"2024-03-11T18:31:29.580546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:34.272568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610416+0000","last_change":"2024-03-11T18:30:28.067822+0000","last_active":"2024-03-11T18:31:29.610416+0000","last_peered":"2024-03-11T18:31:29.610416+0000","last_clean":"2024-03-11T18:31:29.610416+0000","last_became_active":"2024-03-11T18:30:28.067379+0000","last_became_peered":"2024-03-11T18:30:28.067379+0000","last_unstale":"2024-03-11T18:31:29.610416+0000","last_undegraded":"2024-03-11T18:31:29.610416+0000","last_fullsized":"2024-03-11T18:31:29.610416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:27.730186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584814+0000","last_change":"2024-03-11T18:30:32.527845+0000","last_active":"2024-03-11T18:31:29.584814+0000","last_peered":"2024-03-11T18:31:29.584814+0000","last_clean":"2024-03-11T18:31:29.584814+0000","last_became_active":"2024-03-11T18:30:32.527356+0000","last_became_peered":"2024-03-11T18:30:32.527356+0000","last_unstale":"2024-03-11T18:31:29.584814+0000","last_undegraded":"2024-03-11T18:31:29.584814+0000","last_fullsized":"2024-03-11T18:31:29.584814+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.005118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610309+0000","last_change":"2024-03-11T18:30:27.267263+0000","last_active":"2024-03-11T18:31:29.610309+0000","last_peered":"2024-03-11T18:31:29.610309+0000","last_clean":"2024-03-11T18:31:29.610309+0000","last_became_active":"2024-03-11T18:30:27.266893+0000","last_became_peered":"2024-03-11T18:30:27.266893+0000","last_unstale":"2024-03-11T18:31:29.610309+0000","last_undegraded":"2024-03-11T18:31:29.610309+0000","last_fullsized":"2024-03-11T18:31:29.610309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:04.305623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584121+0000","last_change":"2024-03-11T18:30:28.054211+0000","last_active":"2024-03-11T18:31:29.584121+0000","last_peered":"2024-03-11T18:31:29.584121+0000","last_clean":"2024-03-11T18:31:29.584121+0000","last_became_active":"2024-03-11T18:30:28.053472+0000","last_became_peered":"2024-03-11T18:30:28.053472+0000","last_unstale":"2024-03-11T18:31:29.584121+0000","last_undegraded":"2024-03-11T18:31:29.584121+0000","last_fullsized":"2024-03-11T18:31:29.584121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:37.415224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608936+0000","last_change":"2024-03-11T18:30:32.304049+0000","last_active":"2024-03-11T18:31:29.608936+0000","last_peered":"2024-03-11T18:31:29.608936+0000","last_clean":"2024-03-11T18:31:29.608936+0000","last_became_active":"2024-03-11T18:30:32.303410+0000","last_became_peered":"2024-03-11T18:30:32.303410+0000","last_unstale":"2024-03-11T18:31:29.608936+0000","last_undegraded":"2024-03-11T18:31:29.608936+0000","last_fullsized":"2024-03-11T18:31:29.608936+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:15.574274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581052+0000","last_change":"2024-03-11T18:30:27.263269+0000","last_active":"2024-03-11T18:31:29.581052+0000","last_peered":"2024-03-11T18:31:29.581052+0000","last_clean":"2024-03-11T18:31:29.581052+0000","last_became_active":"2024-03-11T18:30:27.262892+0000","last_became_peered":"2024-03-11T18:30:27.262892+0000","last_unstale":"2024-03-11T18:31:29.581052+0000","last_undegraded":"2024-03-11T18:31:29.581052+0000","last_fullsized":"2024-03-11T18:31:29.581052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:59.753276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584750+0000","last_change":"2024-03-11T18:30:28.043362+0000","last_active":"2024-03-11T18:31:29.584750+0000","last_peered":"2024-03-11T18:31:29.584750+0000","last_clean":"2024-03-11T18:31:29.584750+0000","last_became_active":"2024-03-11T18:30:28.042566+0000","last_became_peered":"2024-03-11T18:30:28.042566+0000","last_unstale":"2024-03-11T18:31:29.584750+0000","last_undegraded":"2024-03-11T18:31:29.584750+0000","last_fullsized":"2024-03-11T18:31:29.584750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:41.226647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.612747+0000","last_change":"2024-03-11T18:31:29.612747+0000","last_active":"2024-03-11T18:31:29.612747+0000","last_peered":"2024-03-11T18:31:29.612747+0000","last_clean":"2024-03-11T18:31:29.612747+0000","last_became_active":"2024-03-11T18:31:29.611916+0000","last_became_peered":"2024-03-11T18:31:29.611916+0000","last_unstale":"2024-03-11T18:31:29.612747+0000","last_undegraded":"2024-03-11T18:31:29.612747+0000","last_fullsized":"2024-03-11T18:31:29.612747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:16.349623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579479+0000","last_change":"2024-03-11T18:30:27.271415+0000","last_active":"2024-03-11T18:31:29.579479+0000","last_peered":"2024-03-11T18:31:29.579479+0000","last_clean":"2024-03-11T18:31:29.579479+0000","last_became_active":"2024-03-11T18:30:27.271041+0000","last_became_peered":"2024-03-11T18:30:27.271041+0000","last_unstale":"2024-03-11T18:31:29.579479+0000","last_undegraded":"2024-03-11T18:31:29.579479+0000","last_fullsized":"2024-03-11T18:31:29.579479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:06.967984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.593827+0000","last_change":"2024-03-11T18:31:29.593827+0000","last_active":"2024-03-11T18:31:29.593827+0000","last_peered":"2024-03-11T18:31:29.593827+0000","last_clean":"2024-03-11T18:31:29.593827+0000","last_became_active":"2024-03-11T18:31:29.593485+0000","last_became_peered":"2024-03-11T18:31:29.593485+0000","last_unstale":"2024-03-11T18:31:29.593827+0000","last_undegraded":"2024-03-11T18:31:29.593827+0000","last_fullsized":"2024-03-11T18:31:29.593827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:24:27.782987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578619+0000","last_change":"2024-03-11T18:30:32.304111+0000","last_active":"2024-03-11T18:31:29.578619+0000","last_peered":"2024-03-11T18:31:29.578619+0000","last_clean":"2024-03-11T18:31:29.578619+0000","last_became_active":"2024-03-11T18:30:32.303420+0000","last_became_peered":"2024-03-11T18:30:32.303420+0000","last_unstale":"2024-03-11T18:31:29.578619+0000","last_undegraded":"2024-03-11T18:31:29.578619+0000","last_fullsized":"2024-03-11T18:31:29.578619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:51.801772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.582234+0000","last_change":"2024-03-11T18:30:27.288015+0000","last_active":"2024-03-11T18:31:29.582234+0000","last_peered":"2024-03-11T18:31:29.582234+0000","last_clean":"2024-03-11T18:31:29.582234+0000","last_became_active":"2024-03-11T18:30:27.287528+0000","last_became_peered":"2024-03-11T18:30:27.287528+0000","last_unstale":"2024-03-11T18:31:29.582234+0000","last_undegraded":"2024-03-11T18:31:29.582234+0000","last_fullsized":"2024-03-11T18:31:29.582234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:56:33.612120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583592+0000","last_change":"2024-03-11T18:30:28.041888+0000","last_active":"2024-03-11T18:31:29.583592+0000","last_peered":"2024-03-11T18:31:29.583592+0000","last_clean":"2024-03-11T18:31:29.583592+0000","last_became_active":"2024-03-11T18:30:28.041506+0000","last_became_peered":"2024-03-11T18:30:28.041506+0000","last_unstale":"2024-03-11T18:31:29.583592+0000","last_undegraded":"2024-03-11T18:31:29.583592+0000","last_fullsized":"2024-03-11T18:31:29.583592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:46.308525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586241+0000","last_change":"2024-03-11T18:30:28.037778+0000","last_active":"2024-03-11T18:31:29.586241+0000","last_peered":"2024-03-11T18:31:29.586241+0000","last_clean":"2024-03-11T18:31:29.586241+0000","last_became_active":"2024-03-11T18:30:28.037428+0000","last_became_peered":"2024-03-11T18:30:28.037428+0000","last_unstale":"2024-03-11T18:31:29.586241+0000","last_undegraded":"2024-03-11T18:31:29.586241+0000","last_fullsized":"2024-03-11T18:31:29.586241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:30:35.810425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585187+0000","last_change":"2024-03-11T18:31:29.585187+0000","last_active":"2024-03-11T18:31:29.585187+0000","last_peered":"2024-03-11T18:31:29.585187+0000","last_clean":"2024-03-11T18:31:29.585187+0000","last_became_active":"2024-03-11T18:31:29.584654+0000","last_became_peered":"2024-03-11T18:31:29.584654+0000","last_unstale":"2024-03-11T18:31:29.585187+0000","last_undegraded":"2024-03-11T18:31:29.585187+0000","last_fullsized":"2024-03-11T18:31:29.585187+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:07.081275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580499+0000","last_change":"2024-03-11T18:30:28.060947+0000","last_active":"2024-03-11T18:31:29.580499+0000","last_peered":"2024-03-11T18:31:29.580499+0000","last_clean":"2024-03-11T18:31:29.580499+0000","last_became_active":"2024-03-11T18:30:28.060600+0000","last_became_peered":"2024-03-11T18:30:28.060600+0000","last_unstale":"2024-03-11T18:31:29.580499+0000","last_undegraded":"2024-03-11T18:31:29.580499+0000","last_fullsized":"2024-03-11T18:31:29.580499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:31.964347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594893+0000","last_change":"2024-03-11T18:30:28.062110+0000","last_active":"2024-03-11T18:31:29.594893+0000","last_peered":"2024-03-11T18:31:29.594893+0000","last_clean":"2024-03-11T18:31:29.594893+0000","last_became_active":"2024-03-11T18:30:28.061427+0000","last_became_peered":"2024-03-11T18:30:28.061427+0000","last_unstale":"2024-03-11T18:31:29.594893+0000","last_undegraded":"2024-03-11T18:31:29.594893+0000","last_fullsized":"2024-03-11T18:31:29.594893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:32.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.598122+0000","last_change":"2024-03-11T18:31:29.598122+0000","last_active":"2024-03-11T18:31:29.598122+0000","last_peered":"2024-03-11T18:31:29.598122+0000","last_clean":"2024-03-11T18:31:29.598122+0000","last_became_active":"2024-03-11T18:31:29.597369+0000","last_became_peered":"2024-03-11T18:31:29.597369+0000","last_unstale":"2024-03-11T18:31:29.598122+0000","last_undegraded":"2024-03-11T18:31:29.598122+0000","last_fullsized":"2024-03-11T18:31:29.598122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:46:12.579516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.587042+0000","last_change":"2024-03-11T18:30:28.041817+0000","last_active":"2024-03-11T18:31:29.587042+0000","last_peered":"2024-03-11T18:31:29.587042+0000","last_clean":"2024-03-11T18:31:29.587042+0000","last_became_active":"2024-03-11T18:30:28.041346+0000","last_became_peered":"2024-03-11T18:30:28.041346+0000","last_unstale":"2024-03-11T18:31:29.587042+0000","last_undegraded":"2024-03-11T18:31:29.587042+0000","last_fullsized":"2024-03-11T18:31:29.587042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:26:28.544602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581230+0000","last_change":"2024-03-11T18:30:28.040820+0000","last_active":"2024-03-11T18:31:29.581230+0000","last_peered":"2024-03-11T18:31:29.581230+0000","last_clean":"2024-03-11T18:31:29.581230+0000","last_became_active":"2024-03-11T18:30:28.040381+0000","last_became_peered":"2024-03-11T18:30:28.040381+0000","last_unstale":"2024-03-11T18:31:29.581230+0000","last_undegraded":"2024-03-11T18:31:29.581230+0000","last_fullsized":"2024-03-11T18:31:29.581230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:33:37.120786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610760+0000","last_change":"2024-03-11T18:30:28.052050+0000","last_active":"2024-03-11T18:31:29.610760+0000","last_peered":"2024-03-11T18:31:29.610760+0000","last_clean":"2024-03-11T18:31:29.610760+0000","last_became_active":"2024-03-11T18:30:28.051685+0000","last_became_peered":"2024-03-11T18:30:28.051685+0000","last_unstale":"2024-03-11T18:31:29.610760+0000","last_undegraded":"2024-03-11T18:31:29.610760+0000","last_fullsized":"2024-03-11T18:31:29.610760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:02.777603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579082+0000","last_change":"2024-03-11T18:30:28.053993+0000","last_active":"2024-03-11T18:31:29.579082+0000","last_peered":"2024-03-11T18:31:29.579082+0000","last_clean":"2024-03-11T18:31:29.579082+0000","last_became_active":"2024-03-11T18:30:28.053678+0000","last_became_peered":"2024-03-11T18:30:28.053678+0000","last_unstale":"2024-03-11T18:31:29.579082+0000","last_undegraded":"2024-03-11T18:31:29.579082+0000","last_fullsized":"2024-03-11T18:31:29.579082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:48.801078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609622+0000","last_change":"2024-03-11T18:30:28.051357+0000","last_active":"2024-03-11T18:31:29.609622+0000","last_peered":"2024-03-11T18:31:29.609622+0000","last_clean":"2024-03-11T18:31:29.609622+0000","last_became_active":"2024-03-11T18:30:28.051046+0000","last_became_peered":"2024-03-11T18:30:28.051046+0000","last_unstale":"2024-03-11T18:31:29.609622+0000","last_undegraded":"2024-03-11T18:31:29.609622+0000","last_fullsized":"2024-03-11T18:31:29.609622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:11.994754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609246+0000","last_change":"2024-03-11T18:30:28.051628+0000","last_active":"2024-03-11T18:31:29.609246+0000","last_peered":"2024-03-11T18:31:29.609246+0000","last_clean":"2024-03-11T18:31:29.609246+0000","last_became_active":"2024-03-11T18:30:28.051129+0000","last_became_peered":"2024-03-11T18:30:28.051129+0000","last_unstale":"2024-03-11T18:31:29.609246+0000","last_undegraded":"2024-03-11T18:31:29.609246+0000","last_fullsized":"2024-03-11T18:31:29.609246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:47.568747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581147+0000","last_change":"2024-03-11T18:30:28.065792+0000","last_active":"2024-03-11T18:31:29.581147+0000","last_peered":"2024-03-11T18:31:29.581147+0000","last_clean":"2024-03-11T18:31:29.581147+0000","last_became_active":"2024-03-11T18:30:28.064902+0000","last_became_peered":"2024-03-11T18:30:28.064902+0000","last_unstale":"2024-03-11T18:31:29.581147+0000","last_undegraded":"2024-03-11T18:31:29.581147+0000","last_fullsized":"2024-03-11T18:31:29.581147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:18.019872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586665+0000","last_change":"2024-03-11T18:30:28.041663+0000","last_active":"2024-03-11T18:31:29.586665+0000","last_peered":"2024-03-11T18:31:29.586665+0000","last_clean":"2024-03-11T18:31:29.586665+0000","last_became_active":"2024-03-11T18:30:28.041257+0000","last_became_peered":"2024-03-11T18:30:28.041257+0000","last_unstale":"2024-03-11T18:31:29.586665+0000","last_undegraded":"2024-03-11T18:31:29.586665+0000","last_fullsized":"2024-03-11T18:31:29.586665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:18.216295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67748,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":497,"num_read_kb":322,"num_write":166,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":203244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318308,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":507441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28716,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716436,"statfs":{"total":95995035648,"available":95965630464,"internally_reserved":0,"allocated":1736704,"data_stored":1162552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":511773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005110,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":507597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1103897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430273,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":502853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":566119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":504241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":566847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117074,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":503047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855437,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":508907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691911,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1094998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T18:32:26.891 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T18:32:27.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:26 smithi165 ceph-mon[36634]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:27.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:26 smithi023 ceph-mon[36406]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:27.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:26 smithi002 ceph-mon[31328]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:27.367 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:27 smithi165 ceph-mon[36634]: pgmap v382: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:27 smithi023 ceph-mon[36406]: pgmap v382: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:28.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:27 smithi002 ceph-mon[31328]: pgmap v382: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:29.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-11T18:32:29.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-11T18:32:29.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-11T18:32:29.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:28 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-11T18:32:29.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-11T18:32:29.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-11T18:32:29.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-11T18:32:29.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:28 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-11T18:32:29.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-11T18:32:29.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-11T18:32:29.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-11T18:32:29.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:28 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-11T18:32:29.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 -- 172.21.15.2:0/2522272414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff42410e670 msgr2=0x7ff424110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 --2- 172.21.15.2:0/2522272414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff42410e670 0x7ff424110a60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00b0a0 tx=0x7ff41c00b3b0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 -- 172.21.15.2:0/2522272414 shutdown_connections 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 --2- 172.21.15.2:0/2522272414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff424073ea0 0x7ff424074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 --2- 172.21.15.2:0/2522272414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff42410e670 0x7ff424110a60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 --2- 172.21.15.2:0/2522272414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42410e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 -- 172.21.15.2:0/2522272414 >> 172.21.15.2:0/2522272414 conn(0x7ff424078dd0 msgr2=0x7ff4240791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 -- 172.21.15.2:0/2522272414 shutdown_connections 2024-03-11T18:32:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.511+0000 7ff429e00700 1 -- 172.21.15.2:0/2522272414 wait complete. 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 Processor -- start 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 -- start start 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 0x7ff42419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:29.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419f400 con 0x7ff424073ea0 2024-03-11T18:32:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419f570 con 0x7ff4240747c0 2024-03-11T18:32:29.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.513+0000 7ff429e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42419f730 con 0x7ff42410e670 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff423400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 0x7ff42419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff423400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39186/0 (socket says 172.21.15.2:39186) 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51412/0 (socket says 172.21.15.2:51412) 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff423400700 1 -- 172.21.15.2:0/3889307104 learned_addr learned my addr 172.21.15.2:0/3889307104 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 -- 172.21.15.2:0/3889307104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 msgr2=0x7ff42419edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 -- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 msgr2=0x7ff42419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 0x7ff42419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422000700 1 -- 172.21.15.2:0/3889307104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410009520 con 0x7ff4240747c0 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff423400700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.514+0000 7ff422a00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 0x7ff42419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff422000700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff41000e9f0 tx=0x7ff41000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41000cb70 con 0x7ff4240747c0 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c009d00 con 0x7ff4240747c0 2024-03-11T18:32:29.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff41000ee90 con 0x7ff4240747c0 2024-03-11T18:32:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410003d80 con 0x7ff4240747c0 2024-03-11T18:32:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.515+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4241a70a0 con 0x7ff4240747c0 2024-03-11T18:32:29.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.516+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff424068930 con 0x7ff4240747c0 2024-03-11T18:32:29.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.518+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7ff410003ee0 con 0x7ff4240747c0 2024-03-11T18:32:29.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.520+0000 7ff428a00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 0x7ff4080a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.520+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ff410014070 con 0x7ff4240747c0 2024-03-11T18:32:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.520+0000 7ff422a00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 0x7ff4080a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:29.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.521+0000 7ff422a00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 0x7ff4080a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff414011490 tx=0x7ff414009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.524+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4100968d0 con 0x7ff4240747c0 2024-03-11T18:32:29.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.745+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff424113fa0 con 0x7ff4080a1ac0 2024-03-11T18:32:29.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.763+0000 7ff428a00700 1 -- 172.21.15.2:0/3889307104 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340394 (secure 0 0 0) 0x7ff424113fa0 con 0x7ff4080a1ac0 2024-03-11T18:32:29.767 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:29.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 msgr2=0x7ff4080a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:29.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 0x7ff4080a3f80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff414011490 tx=0x7ff414009260 comp rx=0 tx=0).stop 2024-03-11T18:32:29.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 msgr2=0x7ff42419aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:29.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff41000e9f0 tx=0x7ff41000ed00 comp rx=0 tx=0).stop 2024-03-11T18:32:29.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 shutdown_connections 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4080a1ac0 0x7ff4080a3f80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff42410e670 0x7ff42419edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff424073ea0 0x7ff42419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 --2- 172.21.15.2:0/3889307104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4240747c0 0x7ff42419aab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 >> 172.21.15.2:0/3889307104 conn(0x7ff424078dd0 msgr2=0x7ff4240fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:29.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 shutdown_connections 2024-03-11T18:32:29.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:29.770+0000 7ff429e00700 1 -- 172.21.15.2:0/3889307104 wait complete. 2024-03-11T18:32:29.779 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T18:32:30.208 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:29 smithi002 ceph-mon[31328]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T18:32:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-11T18:32:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-11T18:32:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-11T18:32:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:29 smithi023 ceph-mon[36406]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: pgmap v383: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-11T18:32:30.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-11T18:32:30.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: osdmap e94: 12 total, 12 up, 12 in 2024-03-11T18:32:30.408 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:29 smithi165 ceph-mon[36634]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-11T18:32:30.608 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-03-11T18:32:28.902169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658350,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":663,"num_read_kb":456,"num_write":349,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1181866,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367668,"kb_used_data":14288,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574156,"statfs":{"total":1151940427776,"available":1151563935744,"internally_reserved":0,"allocated":14630912,"data_stored":8040272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.655621"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586747+0000","last_change":"2024-03-11T18:30:28.060909+0000","last_active":"2024-03-11T18:31:29.586747+0000","last_peered":"2024-03-11T18:31:29.586747+0000","last_clean":"2024-03-11T18:31:29.586747+0000","last_became_active":"2024-03-11T18:30:28.060554+0000","last_became_peered":"2024-03-11T18:30:28.060554+0000","last_unstale":"2024-03-11T18:31:29.586747+0000","last_undegraded":"2024-03-11T18:31:29.586747+0000","last_fullsized":"2024-03-11T18:31:29.586747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:11:33.609820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591731+0000","last_change":"2024-03-11T18:30:28.055136+0000","last_active":"2024-03-11T18:31:29.591731+0000","last_peered":"2024-03-11T18:31:29.591731+0000","last_clean":"2024-03-11T18:31:29.591731+0000","last_became_active":"2024-03-11T18:30:28.054785+0000","last_became_peered":"2024-03-11T18:30:28.054785+0000","last_unstale":"2024-03-11T18:31:29.591731+0000","last_undegraded":"2024-03-11T18:31:29.591731+0000","last_fullsized":"2024-03-11T18:31:29.591731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:08:12.702850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590006+0000","last_change":"2024-03-11T18:30:28.051593+0000","last_active":"2024-03-11T18:31:29.590006+0000","last_peered":"2024-03-11T18:31:29.590006+0000","last_clean":"2024-03-11T18:31:29.590006+0000","last_became_active":"2024-03-11T18:30:28.049344+0000","last_became_peered":"2024-03-11T18:30:28.049344+0000","last_unstale":"2024-03-11T18:31:29.590006+0000","last_undegraded":"2024-03-11T18:31:29.590006+0000","last_fullsized":"2024-03-11T18:31:29.590006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:14:06.433202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585364+0000","last_change":"2024-03-11T18:30:28.036335+0000","last_active":"2024-03-11T18:31:29.585364+0000","last_peered":"2024-03-11T18:31:29.585364+0000","last_clean":"2024-03-11T18:31:29.585364+0000","last_became_active":"2024-03-11T18:30:28.035728+0000","last_became_peered":"2024-03-11T18:30:28.035728+0000","last_unstale":"2024-03-11T18:31:29.585364+0000","last_undegraded":"2024-03-11T18:31:29.585364+0000","last_fullsized":"2024-03-11T18:31:29.585364+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:55:10.728266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610339+0000","last_change":"2024-03-11T18:30:28.050560+0000","last_active":"2024-03-11T18:31:29.610339+0000","last_peered":"2024-03-11T18:31:29.610339+0000","last_clean":"2024-03-11T18:31:29.610339+0000","last_became_active":"2024-03-11T18:30:28.050273+0000","last_became_peered":"2024-03-11T18:30:28.050273+0000","last_unstale":"2024-03-11T18:31:29.610339+0000","last_undegraded":"2024-03-11T18:31:29.610339+0000","last_fullsized":"2024-03-11T18:31:29.610339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:35:56.780575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609873+0000","last_change":"2024-03-11T18:30:28.056879+0000","last_active":"2024-03-11T18:31:29.609873+0000","last_peered":"2024-03-11T18:31:29.609873+0000","last_clean":"2024-03-11T18:31:29.609873+0000","last_became_active":"2024-03-11T18:30:28.056435+0000","last_became_peered":"2024-03-11T18:30:28.056435+0000","last_unstale":"2024-03-11T18:31:29.609873+0000","last_undegraded":"2024-03-11T18:31:29.609873+0000","last_fullsized":"2024-03-11T18:31:29.609873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:04.595503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584088+0000","last_change":"2024-03-11T18:30:28.044355+0000","last_active":"2024-03-11T18:31:29.584088+0000","last_peered":"2024-03-11T18:31:29.584088+0000","last_clean":"2024-03-11T18:31:29.584088+0000","last_became_active":"2024-03-11T18:30:28.043875+0000","last_became_peered":"2024-03-11T18:30:28.043875+0000","last_unstale":"2024-03-11T18:31:29.584088+0000","last_undegraded":"2024-03-11T18:31:29.584088+0000","last_fullsized":"2024-03-11T18:31:29.584088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:42:25.924359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609458+0000","last_change":"2024-03-11T18:30:28.055633+0000","last_active":"2024-03-11T18:31:29.609458+0000","last_peered":"2024-03-11T18:31:29.609458+0000","last_clean":"2024-03-11T18:31:29.609458+0000","last_became_active":"2024-03-11T18:30:28.055158+0000","last_became_peered":"2024-03-11T18:30:28.055158+0000","last_unstale":"2024-03-11T18:31:29.609458+0000","last_undegraded":"2024-03-11T18:31:29.609458+0000","last_fullsized":"2024-03-11T18:31:29.609458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:21:45.379551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609345+0000","last_change":"2024-03-11T18:30:28.037306+0000","last_active":"2024-03-11T18:31:29.609345+0000","last_peered":"2024-03-11T18:31:29.609345+0000","last_clean":"2024-03-11T18:31:29.609345+0000","last_became_active":"2024-03-11T18:30:28.036930+0000","last_became_peered":"2024-03-11T18:30:28.036930+0000","last_unstale":"2024-03-11T18:31:29.609345+0000","last_undegraded":"2024-03-11T18:31:29.609345+0000","last_fullsized":"2024-03-11T18:31:29.609345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610002+0000","last_change":"2024-03-11T18:30:28.038914+0000","last_active":"2024-03-11T18:31:29.610002+0000","last_peered":"2024-03-11T18:31:29.610002+0000","last_clean":"2024-03-11T18:31:29.610002+0000","last_became_active":"2024-03-11T18:30:28.038266+0000","last_became_peered":"2024-03-11T18:30:28.038266+0000","last_unstale":"2024-03-11T18:31:29.610002+0000","last_undegraded":"2024-03-11T18:31:29.610002+0000","last_fullsized":"2024-03-11T18:31:29.610002+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:59.136860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584650+0000","last_change":"2024-03-11T18:30:28.041833+0000","last_active":"2024-03-11T18:31:29.584650+0000","last_peered":"2024-03-11T18:31:29.584650+0000","last_clean":"2024-03-11T18:31:29.584650+0000","last_became_active":"2024-03-11T18:30:28.041307+0000","last_became_peered":"2024-03-11T18:30:28.041307+0000","last_unstale":"2024-03-11T18:31:29.584650+0000","last_undegraded":"2024-03-11T18:31:29.584650+0000","last_fullsized":"2024-03-11T18:31:29.584650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:55:35.803202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585230+0000","last_change":"2024-03-11T18:30:28.037646+0000","last_active":"2024-03-11T18:31:29.585230+0000","last_peered":"2024-03-11T18:31:29.585230+0000","last_clean":"2024-03-11T18:31:29.585230+0000","last_became_active":"2024-03-11T18:30:28.037272+0000","last_became_peered":"2024-03-11T18:30:28.037272+0000","last_unstale":"2024-03-11T18:31:29.585230+0000","last_undegraded":"2024-03-11T18:31:29.585230+0000","last_fullsized":"2024-03-11T18:31:29.585230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:50.594666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580926+0000","last_change":"2024-03-11T18:30:28.056917+0000","last_active":"2024-03-11T18:31:29.580926+0000","last_peered":"2024-03-11T18:31:29.580926+0000","last_clean":"2024-03-11T18:31:29.580926+0000","last_became_active":"2024-03-11T18:30:28.056496+0000","last_became_peered":"2024-03-11T18:30:28.056496+0000","last_unstale":"2024-03-11T18:31:29.580926+0000","last_undegraded":"2024-03-11T18:31:29.580926+0000","last_fullsized":"2024-03-11T18:31:29.580926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:44.125228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594231+0000","last_change":"2024-03-11T18:30:28.055180+0000","last_active":"2024-03-11T18:31:29.594231+0000","last_peered":"2024-03-11T18:31:29.594231+0000","last_clean":"2024-03-11T18:31:29.594231+0000","last_became_active":"2024-03-11T18:30:28.054807+0000","last_became_peered":"2024-03-11T18:30:28.054807+0000","last_unstale":"2024-03-11T18:31:29.594231+0000","last_undegraded":"2024-03-11T18:31:29.594231+0000","last_fullsized":"2024-03-11T18:31:29.594231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:31.671461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590883+0000","last_change":"2024-03-11T18:30:28.065015+0000","last_active":"2024-03-11T18:31:29.590883+0000","last_peered":"2024-03-11T18:31:29.590883+0000","last_clean":"2024-03-11T18:31:29.590883+0000","last_became_active":"2024-03-11T18:30:28.061131+0000","last_became_peered":"2024-03-11T18:30:28.061131+0000","last_unstale":"2024-03-11T18:31:29.590883+0000","last_undegraded":"2024-03-11T18:31:29.590883+0000","last_fullsized":"2024-03-11T18:31:29.590883+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:13:35.314562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608751+0000","last_change":"2024-03-11T18:30:28.054388+0000","last_active":"2024-03-11T18:31:29.608751+0000","last_peered":"2024-03-11T18:31:29.608751+0000","last_clean":"2024-03-11T18:31:29.608751+0000","last_became_active":"2024-03-11T18:30:28.054036+0000","last_became_peered":"2024-03-11T18:30:28.054036+0000","last_unstale":"2024-03-11T18:31:29.608751+0000","last_undegraded":"2024-03-11T18:31:29.608751+0000","last_fullsized":"2024-03-11T18:31:29.608751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:45:21.476944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586710+0000","last_change":"2024-03-11T18:30:28.061935+0000","last_active":"2024-03-11T18:31:29.586710+0000","last_peered":"2024-03-11T18:31:29.586710+0000","last_clean":"2024-03-11T18:31:29.586710+0000","last_became_active":"2024-03-11T18:30:28.061558+0000","last_became_peered":"2024-03-11T18:30:28.061558+0000","last_unstale":"2024-03-11T18:31:29.586710+0000","last_undegraded":"2024-03-11T18:31:29.586710+0000","last_fullsized":"2024-03-11T18:31:29.586710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:05:01.728956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594476+0000","last_change":"2024-03-11T18:30:28.058372+0000","last_active":"2024-03-11T18:31:29.594476+0000","last_peered":"2024-03-11T18:31:29.594476+0000","last_clean":"2024-03-11T18:31:29.594476+0000","last_became_active":"2024-03-11T18:30:28.058028+0000","last_became_peered":"2024-03-11T18:30:28.058028+0000","last_unstale":"2024-03-11T18:31:29.594476+0000","last_undegraded":"2024-03-11T18:31:29.594476+0000","last_fullsized":"2024-03-11T18:31:29.594476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:20:15.645368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580740+0000","last_change":"2024-03-11T18:30:28.054845+0000","last_active":"2024-03-11T18:31:29.580740+0000","last_peered":"2024-03-11T18:31:29.580740+0000","last_clean":"2024-03-11T18:31:29.580740+0000","last_became_active":"2024-03-11T18:30:28.054497+0000","last_became_peered":"2024-03-11T18:30:28.054497+0000","last_unstale":"2024-03-11T18:31:29.580740+0000","last_undegraded":"2024-03-11T18:31:29.580740+0000","last_fullsized":"2024-03-11T18:31:29.580740+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:52.100045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580986+0000","last_change":"2024-03-11T18:30:32.322103+0000","last_active":"2024-03-11T18:31:29.580986+0000","last_peered":"2024-03-11T18:31:29.580986+0000","last_clean":"2024-03-11T18:31:29.580986+0000","last_became_active":"2024-03-11T18:30:32.321422+0000","last_became_peered":"2024-03-11T18:30:32.321422+0000","last_unstale":"2024-03-11T18:31:29.580986+0000","last_undegraded":"2024-03-11T18:31:29.580986+0000","last_fullsized":"2024-03-11T18:31:29.580986+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:56.946318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.596373+0000","last_change":"2024-03-11T18:30:27.283463+0000","last_active":"2024-03-11T18:31:29.596373+0000","last_peered":"2024-03-11T18:31:29.596373+0000","last_clean":"2024-03-11T18:31:29.596373+0000","last_became_active":"2024-03-11T18:30:27.279100+0000","last_became_peered":"2024-03-11T18:30:27.279100+0000","last_unstale":"2024-03-11T18:31:29.596373+0000","last_undegraded":"2024-03-11T18:31:29.596373+0000","last_fullsized":"2024-03-11T18:31:29.596373+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:40:38.096395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610504+0000","last_change":"2024-03-11T18:30:28.051288+0000","last_active":"2024-03-11T18:31:29.610504+0000","last_peered":"2024-03-11T18:31:29.610504+0000","last_clean":"2024-03-11T18:31:29.610504+0000","last_became_active":"2024-03-11T18:30:28.050726+0000","last_became_peered":"2024-03-11T18:30:28.050726+0000","last_unstale":"2024-03-11T18:31:29.610504+0000","last_undegraded":"2024-03-11T18:31:29.610504+0000","last_fullsized":"2024-03-11T18:31:29.610504+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:29.125099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584976+0000","last_change":"2024-03-11T18:30:32.311188+0000","last_active":"2024-03-11T18:31:29.584976+0000","last_peered":"2024-03-11T18:31:29.584976+0000","last_clean":"2024-03-11T18:31:29.584976+0000","last_became_active":"2024-03-11T18:30:32.310670+0000","last_became_peered":"2024-03-11T18:30:32.310670+0000","last_unstale":"2024-03-11T18:31:29.584976+0000","last_undegraded":"2024-03-11T18:31:29.584976+0000","last_fullsized":"2024-03-11T18:31:29.584976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:51:17.810402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.611088+0000","last_change":"2024-03-11T18:30:27.284990+0000","last_active":"2024-03-11T18:31:29.611088+0000","last_peered":"2024-03-11T18:31:29.611088+0000","last_clean":"2024-03-11T18:31:29.611088+0000","last_became_active":"2024-03-11T18:30:27.283629+0000","last_became_peered":"2024-03-11T18:30:27.283629+0000","last_unstale":"2024-03-11T18:31:29.611088+0000","last_undegraded":"2024-03-11T18:31:29.611088+0000","last_fullsized":"2024-03-11T18:31:29.611088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:01:33.705606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591401+0000","last_change":"2024-03-11T18:30:28.066345+0000","last_active":"2024-03-11T18:31:29.591401+0000","last_peered":"2024-03-11T18:31:29.591401+0000","last_clean":"2024-03-11T18:31:29.591401+0000","last_became_active":"2024-03-11T18:30:28.061654+0000","last_became_peered":"2024-03-11T18:30:28.061654+0000","last_unstale":"2024-03-11T18:31:29.591401+0000","last_undegraded":"2024-03-11T18:31:29.591401+0000","last_fullsized":"2024-03-11T18:31:29.591401+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:29.651510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585437+0000","last_change":"2024-03-11T18:30:32.318088+0000","last_active":"2024-03-11T18:31:29.585437+0000","last_peered":"2024-03-11T18:31:29.585437+0000","last_clean":"2024-03-11T18:31:29.585437+0000","last_became_active":"2024-03-11T18:30:32.317304+0000","last_became_peered":"2024-03-11T18:30:32.317304+0000","last_unstale":"2024-03-11T18:31:29.585437+0000","last_undegraded":"2024-03-11T18:31:29.585437+0000","last_fullsized":"2024-03-11T18:31:29.585437+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:41:01.678144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610319+0000","last_change":"2024-03-11T18:30:27.268370+0000","last_active":"2024-03-11T18:31:29.610319+0000","last_peered":"2024-03-11T18:31:29.610319+0000","last_clean":"2024-03-11T18:31:29.610319+0000","last_became_active":"2024-03-11T18:30:27.268033+0000","last_became_peered":"2024-03-11T18:30:27.268033+0000","last_unstale":"2024-03-11T18:31:29.610319+0000","last_undegraded":"2024-03-11T18:31:29.610319+0000","last_fullsized":"2024-03-11T18:31:29.610319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:31:13.545893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608975+0000","last_change":"2024-03-11T18:30:28.051611+0000","last_active":"2024-03-11T18:31:29.608975+0000","last_peered":"2024-03-11T18:31:29.608975+0000","last_clean":"2024-03-11T18:31:29.608975+0000","last_became_active":"2024-03-11T18:30:28.051249+0000","last_became_peered":"2024-03-11T18:30:28.051249+0000","last_unstale":"2024-03-11T18:31:29.608975+0000","last_undegraded":"2024-03-11T18:31:29.608975+0000","last_fullsized":"2024-03-11T18:31:29.608975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:12.942910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584639+0000","last_change":"2024-03-11T18:30:32.318934+0000","last_active":"2024-03-11T18:31:29.584639+0000","last_peered":"2024-03-11T18:31:29.584639+0000","last_clean":"2024-03-11T18:31:29.584639+0000","last_became_active":"2024-03-11T18:30:32.318474+0000","last_became_peered":"2024-03-11T18:30:32.318474+0000","last_unstale":"2024-03-11T18:31:29.584639+0000","last_undegraded":"2024-03-11T18:31:29.584639+0000","last_fullsized":"2024-03-11T18:31:29.584639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:48:33.571185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610073+0000","last_change":"2024-03-11T18:30:27.268675+0000","last_active":"2024-03-11T18:31:29.610073+0000","last_peered":"2024-03-11T18:31:29.610073+0000","last_clean":"2024-03-11T18:31:29.610073+0000","last_became_active":"2024-03-11T18:30:27.268320+0000","last_became_peered":"2024-03-11T18:30:27.268320+0000","last_unstale":"2024-03-11T18:31:29.610073+0000","last_undegraded":"2024-03-11T18:31:29.610073+0000","last_fullsized":"2024-03-11T18:31:29.610073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:06.439930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579354+0000","last_change":"2024-03-11T18:30:28.038380+0000","last_active":"2024-03-11T18:31:29.579354+0000","last_peered":"2024-03-11T18:31:29.579354+0000","last_clean":"2024-03-11T18:31:29.579354+0000","last_became_active":"2024-03-11T18:30:28.037878+0000","last_became_peered":"2024-03-11T18:30:28.037878+0000","last_unstale":"2024-03-11T18:31:29.579354+0000","last_undegraded":"2024-03-11T18:31:29.579354+0000","last_fullsized":"2024-03-11T18:31:29.579354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:10.988295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580594+0000","last_change":"2024-03-11T18:30:32.304747+0000","last_active":"2024-03-11T18:31:29.580594+0000","last_peered":"2024-03-11T18:31:29.580594+0000","last_clean":"2024-03-11T18:31:29.580594+0000","last_became_active":"2024-03-11T18:30:32.304282+0000","last_became_peered":"2024-03-11T18:30:32.304282+0000","last_unstale":"2024-03-11T18:31:29.580594+0000","last_undegraded":"2024-03-11T18:31:29.580594+0000","last_fullsized":"2024-03-11T18:31:29.580594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:24:42.283932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":133,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:24.675197+0000","last_change":"2024-03-11T18:31:29.590831+0000","last_active":"2024-03-11T18:32:24.675197+0000","last_peered":"2024-03-11T18:32:24.675197+0000","last_clean":"2024-03-11T18:32:24.675197+0000","last_became_active":"2024-03-11T18:31:29.586525+0000","last_became_peered":"2024-03-11T18:31:29.586525+0000","last_unstale":"2024-03-11T18:32:24.675197+0000","last_undegraded":"2024-03-11T18:32:24.675197+0000","last_fullsized":"2024-03-11T18:32:24.675197+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:25:04.712037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609917+0000","last_change":"2024-03-11T18:30:28.060280+0000","last_active":"2024-03-11T18:31:29.609917+0000","last_peered":"2024-03-11T18:31:29.609917+0000","last_clean":"2024-03-11T18:31:29.609917+0000","last_became_active":"2024-03-11T18:30:28.059929+0000","last_became_peered":"2024-03-11T18:30:28.059929+0000","last_unstale":"2024-03-11T18:31:29.609917+0000","last_undegraded":"2024-03-11T18:31:29.609917+0000","last_fullsized":"2024-03-11T18:31:29.609917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:47:31.995021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580852+0000","last_change":"2024-03-11T18:30:32.321754+0000","last_active":"2024-03-11T18:31:29.580852+0000","last_peered":"2024-03-11T18:31:29.580852+0000","last_clean":"2024-03-11T18:31:29.580852+0000","last_became_active":"2024-03-11T18:30:32.321329+0000","last_became_peered":"2024-03-11T18:30:32.321329+0000","last_unstale":"2024-03-11T18:31:29.580852+0000","last_undegraded":"2024-03-11T18:31:29.580852+0000","last_fullsized":"2024-03-11T18:31:29.580852+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:48.985005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609746+0000","last_change":"2024-03-11T18:30:27.285442+0000","last_active":"2024-03-11T18:31:29.609746+0000","last_peered":"2024-03-11T18:31:29.609746+0000","last_clean":"2024-03-11T18:31:29.609746+0000","last_became_active":"2024-03-11T18:30:27.285127+0000","last_became_peered":"2024-03-11T18:30:27.285127+0000","last_unstale":"2024-03-11T18:31:29.609746+0000","last_undegraded":"2024-03-11T18:31:29.609746+0000","last_fullsized":"2024-03-11T18:31:29.609746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:01.278846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.614546+0000","last_change":"2024-03-11T18:31:29.612985+0000","last_active":"2024-03-11T18:31:29.614546+0000","last_peered":"2024-03-11T18:31:29.614546+0000","last_clean":"2024-03-11T18:31:29.614546+0000","last_became_active":"2024-03-11T18:31:29.612340+0000","last_became_peered":"2024-03-11T18:31:29.612340+0000","last_unstale":"2024-03-11T18:31:29.614546+0000","last_undegraded":"2024-03-11T18:31:29.614546+0000","last_fullsized":"2024-03-11T18:31:29.614546+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:19:14.723344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586642+0000","last_change":"2024-03-11T18:30:32.327126+0000","last_active":"2024-03-11T18:31:29.586642+0000","last_peered":"2024-03-11T18:31:29.586642+0000","last_clean":"2024-03-11T18:31:29.586642+0000","last_became_active":"2024-03-11T18:30:32.326668+0000","last_became_peered":"2024-03-11T18:30:32.326668+0000","last_unstale":"2024-03-11T18:31:29.586642+0000","last_undegraded":"2024-03-11T18:31:29.586642+0000","last_fullsized":"2024-03-11T18:31:29.586642+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:03.231714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580792+0000","last_change":"2024-03-11T18:30:27.289439+0000","last_active":"2024-03-11T18:31:29.580792+0000","last_peered":"2024-03-11T18:31:29.580792+0000","last_clean":"2024-03-11T18:31:29.580792+0000","last_became_active":"2024-03-11T18:30:27.286067+0000","last_became_peered":"2024-03-11T18:30:27.286067+0000","last_unstale":"2024-03-11T18:31:29.580792+0000","last_undegraded":"2024-03-11T18:31:29.580792+0000","last_fullsized":"2024-03-11T18:31:29.580792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:35.522841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609641+0000","last_change":"2024-03-11T18:30:28.058525+0000","last_active":"2024-03-11T18:31:29.609641+0000","last_peered":"2024-03-11T18:31:29.609641+0000","last_clean":"2024-03-11T18:31:29.609641+0000","last_became_active":"2024-03-11T18:30:28.058122+0000","last_became_peered":"2024-03-11T18:30:28.058122+0000","last_unstale":"2024-03-11T18:31:29.609641+0000","last_undegraded":"2024-03-11T18:31:29.609641+0000","last_fullsized":"2024-03-11T18:31:29.609641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:45:04.537880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591536+0000","last_change":"2024-03-11T18:30:32.314757+0000","last_active":"2024-03-11T18:31:29.591536+0000","last_peered":"2024-03-11T18:31:29.591536+0000","last_clean":"2024-03-11T18:31:29.591536+0000","last_became_active":"2024-03-11T18:30:32.313461+0000","last_became_peered":"2024-03-11T18:30:32.313461+0000","last_unstale":"2024-03-11T18:31:29.591536+0000","last_undegraded":"2024-03-11T18:31:29.591536+0000","last_fullsized":"2024-03-11T18:31:29.591536+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:11:27.881788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578835+0000","last_change":"2024-03-11T18:30:27.268359+0000","last_active":"2024-03-11T18:31:29.578835+0000","last_peered":"2024-03-11T18:31:29.578835+0000","last_clean":"2024-03-11T18:31:29.578835+0000","last_became_active":"2024-03-11T18:30:27.268045+0000","last_became_peered":"2024-03-11T18:30:27.268045+0000","last_unstale":"2024-03-11T18:31:29.578835+0000","last_undegraded":"2024-03-11T18:31:29.578835+0000","last_fullsized":"2024-03-11T18:31:29.578835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:49:53.678477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.590413+0000","last_change":"2024-03-11T18:30:28.051331+0000","last_active":"2024-03-11T18:31:29.590413+0000","last_peered":"2024-03-11T18:31:29.590413+0000","last_clean":"2024-03-11T18:31:29.590413+0000","last_became_active":"2024-03-11T18:30:28.050462+0000","last_became_peered":"2024-03-11T18:30:28.050462+0000","last_unstale":"2024-03-11T18:31:29.590413+0000","last_undegraded":"2024-03-11T18:31:29.590413+0000","last_fullsized":"2024-03-11T18:31:29.590413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:24.194650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609940+0000","last_change":"2024-03-11T18:30:32.317173+0000","last_active":"2024-03-11T18:31:29.609940+0000","last_peered":"2024-03-11T18:31:29.609940+0000","last_clean":"2024-03-11T18:31:29.609940+0000","last_became_active":"2024-03-11T18:30:32.316721+0000","last_became_peered":"2024-03-11T18:30:32.316721+0000","last_unstale":"2024-03-11T18:31:29.609940+0000","last_undegraded":"2024-03-11T18:31:29.609940+0000","last_fullsized":"2024-03-11T18:31:29.609940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:54:43.665949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":112,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:22.671623+0000","last_change":"2024-03-11T18:30:27.279586+0000","last_active":"2024-03-11T18:32:22.671623+0000","last_peered":"2024-03-11T18:32:22.671623+0000","last_clean":"2024-03-11T18:32:22.671623+0000","last_became_active":"2024-03-11T18:30:27.279262+0000","last_became_peered":"2024-03-11T18:30:27.279262+0000","last_unstale":"2024-03-11T18:32:22.671623+0000","last_undegraded":"2024-03-11T18:32:22.671623+0000","last_fullsized":"2024-03-11T18:32:22.671623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:02:33.570755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609286+0000","last_change":"2024-03-11T18:30:28.060636+0000","last_active":"2024-03-11T18:31:29.609286+0000","last_peered":"2024-03-11T18:31:29.609286+0000","last_clean":"2024-03-11T18:31:29.609286+0000","last_became_active":"2024-03-11T18:30:28.060282+0000","last_became_peered":"2024-03-11T18:30:28.060282+0000","last_unstale":"2024-03-11T18:31:29.609286+0000","last_undegraded":"2024-03-11T18:31:29.609286+0000","last_fullsized":"2024-03-11T18:31:29.609286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:56.618563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.589505+0000","last_change":"2024-03-11T18:30:32.314680+0000","last_active":"2024-03-11T18:31:29.589505+0000","last_peered":"2024-03-11T18:31:29.589505+0000","last_clean":"2024-03-11T18:31:29.589505+0000","last_became_active":"2024-03-11T18:30:32.314102+0000","last_became_peered":"2024-03-11T18:30:32.314102+0000","last_unstale":"2024-03-11T18:31:29.589505+0000","last_undegraded":"2024-03-11T18:31:29.589505+0000","last_fullsized":"2024-03-11T18:31:29.589505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:18:55.989746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584409+0000","last_change":"2024-03-11T18:30:27.267028+0000","last_active":"2024-03-11T18:31:29.584409+0000","last_peered":"2024-03-11T18:31:29.584409+0000","last_clean":"2024-03-11T18:31:29.584409+0000","last_became_active":"2024-03-11T18:30:27.266712+0000","last_became_peered":"2024-03-11T18:30:27.266712+0000","last_unstale":"2024-03-11T18:31:29.584409+0000","last_undegraded":"2024-03-11T18:31:29.584409+0000","last_fullsized":"2024-03-11T18:31:29.584409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:35.978113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591112+0000","last_change":"2024-03-11T18:30:28.066607+0000","last_active":"2024-03-11T18:31:29.591112+0000","last_peered":"2024-03-11T18:31:29.591112+0000","last_clean":"2024-03-11T18:31:29.591112+0000","last_became_active":"2024-03-11T18:30:28.061181+0000","last_became_peered":"2024-03-11T18:30:28.061181+0000","last_unstale":"2024-03-11T18:31:29.591112+0000","last_undegraded":"2024-03-11T18:31:29.591112+0000","last_fullsized":"2024-03-11T18:31:29.591112+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:30:01.203632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.612447+0000","last_change":"2024-03-11T18:31:29.612447+0000","last_active":"2024-03-11T18:31:29.612447+0000","last_peered":"2024-03-11T18:31:29.612447+0000","last_clean":"2024-03-11T18:31:29.612447+0000","last_became_active":"2024-03-11T18:31:29.611143+0000","last_became_peered":"2024-03-11T18:31:29.611143+0000","last_unstale":"2024-03-11T18:31:29.612447+0000","last_undegraded":"2024-03-11T18:31:29.612447+0000","last_fullsized":"2024-03-11T18:31:29.612447+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:22:23.662676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579244+0000","last_change":"2024-03-11T18:30:27.270147+0000","last_active":"2024-03-11T18:31:29.579244+0000","last_peered":"2024-03-11T18:31:29.579244+0000","last_clean":"2024-03-11T18:31:29.579244+0000","last_became_active":"2024-03-11T18:30:27.269791+0000","last_became_peered":"2024-03-11T18:30:27.269791+0000","last_unstale":"2024-03-11T18:31:29.579244+0000","last_undegraded":"2024-03-11T18:31:29.579244+0000","last_fullsized":"2024-03-11T18:31:29.579244+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:12:22.007518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609088+0000","last_change":"2024-03-11T18:30:28.035357+0000","last_active":"2024-03-11T18:31:29.609088+0000","last_peered":"2024-03-11T18:31:29.609088+0000","last_clean":"2024-03-11T18:31:29.609088+0000","last_became_active":"2024-03-11T18:30:28.035005+0000","last_became_peered":"2024-03-11T18:30:28.035005+0000","last_unstale":"2024-03-11T18:31:29.609088+0000","last_undegraded":"2024-03-11T18:31:29.609088+0000","last_fullsized":"2024-03-11T18:31:29.609088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:21.450440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.589644+0000","last_change":"2024-03-11T18:30:32.323472+0000","last_active":"2024-03-11T18:31:29.589644+0000","last_peered":"2024-03-11T18:31:29.589644+0000","last_clean":"2024-03-11T18:31:29.589644+0000","last_became_active":"2024-03-11T18:30:32.321590+0000","last_became_peered":"2024-03-11T18:30:32.321590+0000","last_unstale":"2024-03-11T18:31:29.589644+0000","last_undegraded":"2024-03-11T18:31:29.589644+0000","last_fullsized":"2024-03-11T18:31:29.589644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:30:12.329127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'33","reported_seq":233,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:23.673270+0000","last_change":"2024-03-11T18:30:27.267683+0000","last_active":"2024-03-11T18:32:23.673270+0000","last_peered":"2024-03-11T18:32:23.673270+0000","last_clean":"2024-03-11T18:32:23.673270+0000","last_became_active":"2024-03-11T18:30:27.267333+0000","last_became_peered":"2024-03-11T18:30:27.267333+0000","last_unstale":"2024-03-11T18:32:23.673270+0000","last_undegraded":"2024-03-11T18:32:23.673270+0000","last_fullsized":"2024-03-11T18:32:23.673270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:33:18.721843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":174,"num_read_kb":149,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'1","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610186+0000","last_change":"2024-03-11T18:30:28.038846+0000","last_active":"2024-03-11T18:31:29.610186+0000","last_peered":"2024-03-11T18:31:29.610186+0000","last_clean":"2024-03-11T18:31:29.610186+0000","last_became_active":"2024-03-11T18:30:28.038479+0000","last_became_peered":"2024-03-11T18:30:28.038479+0000","last_unstale":"2024-03-11T18:31:29.610186+0000","last_undegraded":"2024-03-11T18:31:29.610186+0000","last_fullsized":"2024-03-11T18:31:29.610186+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:39:25.877627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579658+0000","last_change":"2024-03-11T18:30:32.317761+0000","last_active":"2024-03-11T18:31:29.579658+0000","last_peered":"2024-03-11T18:31:29.579658+0000","last_clean":"2024-03-11T18:31:29.579658+0000","last_became_active":"2024-03-11T18:30:32.317438+0000","last_became_peered":"2024-03-11T18:30:32.317438+0000","last_unstale":"2024-03-11T18:31:29.579658+0000","last_undegraded":"2024-03-11T18:31:29.579658+0000","last_fullsized":"2024-03-11T18:31:29.579658+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:43:58.145037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581109+0000","last_change":"2024-03-11T18:30:27.276641+0000","last_active":"2024-03-11T18:31:29.581109+0000","last_peered":"2024-03-11T18:31:29.581109+0000","last_clean":"2024-03-11T18:31:29.581109+0000","last_became_active":"2024-03-11T18:30:27.276293+0000","last_became_peered":"2024-03-11T18:30:27.276293+0000","last_unstale":"2024-03-11T18:31:29.581109+0000","last_undegraded":"2024-03-11T18:31:29.581109+0000","last_fullsized":"2024-03-11T18:31:29.581109+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:46.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580612+0000","last_change":"2024-03-11T18:30:28.064304+0000","last_active":"2024-03-11T18:31:29.580612+0000","last_peered":"2024-03-11T18:31:29.580612+0000","last_clean":"2024-03-11T18:31:29.580612+0000","last_became_active":"2024-03-11T18:30:28.063923+0000","last_became_peered":"2024-03-11T18:30:28.063923+0000","last_unstale":"2024-03-11T18:31:29.580612+0000","last_undegraded":"2024-03-11T18:31:29.580612+0000","last_fullsized":"2024-03-11T18:31:29.580612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:57:41.289717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586465+0000","last_change":"2024-03-11T18:30:32.325409+0000","last_active":"2024-03-11T18:31:29.586465+0000","last_peered":"2024-03-11T18:31:29.586465+0000","last_clean":"2024-03-11T18:31:29.586465+0000","last_became_active":"2024-03-11T18:30:32.324953+0000","last_became_peered":"2024-03-11T18:30:32.324953+0000","last_unstale":"2024-03-11T18:31:29.586465+0000","last_undegraded":"2024-03-11T18:31:29.586465+0000","last_fullsized":"2024-03-11T18:31:29.586465+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:44:54.106119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579334+0000","last_change":"2024-03-11T18:30:27.271690+0000","last_active":"2024-03-11T18:31:29.579334+0000","last_peered":"2024-03-11T18:31:29.579334+0000","last_clean":"2024-03-11T18:31:29.579334+0000","last_became_active":"2024-03-11T18:30:27.271429+0000","last_became_peered":"2024-03-11T18:30:27.271429+0000","last_unstale":"2024-03-11T18:31:29.579334+0000","last_undegraded":"2024-03-11T18:31:29.579334+0000","last_fullsized":"2024-03-11T18:31:29.579334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:55:23.896642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583790+0000","last_change":"2024-03-11T18:30:28.054214+0000","last_active":"2024-03-11T18:31:29.583790+0000","last_peered":"2024-03-11T18:31:29.583790+0000","last_clean":"2024-03-11T18:31:29.583790+0000","last_became_active":"2024-03-11T18:30:28.053447+0000","last_became_peered":"2024-03-11T18:30:28.053447+0000","last_unstale":"2024-03-11T18:31:29.583790+0000","last_undegraded":"2024-03-11T18:31:29.583790+0000","last_fullsized":"2024-03-11T18:31:29.583790+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:08:49.866509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594999+0000","last_change":"2024-03-11T18:30:32.325028+0000","last_active":"2024-03-11T18:31:29.594999+0000","last_peered":"2024-03-11T18:31:29.594999+0000","last_clean":"2024-03-11T18:31:29.594999+0000","last_became_active":"2024-03-11T18:30:32.324526+0000","last_became_peered":"2024-03-11T18:30:32.324526+0000","last_unstale":"2024-03-11T18:31:29.594999+0000","last_undegraded":"2024-03-11T18:31:29.594999+0000","last_fullsized":"2024-03-11T18:31:29.594999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:08.031224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":119,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:25.630911+0000","last_change":"2024-03-11T18:30:27.265284+0000","last_active":"2024-03-11T18:32:25.630911+0000","last_peered":"2024-03-11T18:32:25.630911+0000","last_clean":"2024-03-11T18:32:25.630911+0000","last_became_active":"2024-03-11T18:30:27.264402+0000","last_became_peered":"2024-03-11T18:30:27.264402+0000","last_unstale":"2024-03-11T18:32:25.630911+0000","last_undegraded":"2024-03-11T18:32:25.630911+0000","last_fullsized":"2024-03-11T18:32:25.630911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:34:14.768956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583967+0000","last_change":"2024-03-11T18:30:28.040540+0000","last_active":"2024-03-11T18:31:29.583967+0000","last_peered":"2024-03-11T18:31:29.583967+0000","last_clean":"2024-03-11T18:31:29.583967+0000","last_became_active":"2024-03-11T18:30:28.040181+0000","last_became_peered":"2024-03-11T18:30:28.040181+0000","last_unstale":"2024-03-11T18:31:29.583967+0000","last_undegraded":"2024-03-11T18:31:29.583967+0000","last_fullsized":"2024-03-11T18:31:29.583967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:19.079250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.582353+0000","last_change":"2024-03-11T18:30:32.326182+0000","last_active":"2024-03-11T18:31:29.582353+0000","last_peered":"2024-03-11T18:31:29.582353+0000","last_clean":"2024-03-11T18:31:29.582353+0000","last_became_active":"2024-03-11T18:30:32.325793+0000","last_became_peered":"2024-03-11T18:30:32.325793+0000","last_unstale":"2024-03-11T18:31:29.582353+0000","last_undegraded":"2024-03-11T18:31:29.582353+0000","last_fullsized":"2024-03-11T18:31:29.582353+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:14:13.212511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609646+0000","last_change":"2024-03-11T18:30:27.266724+0000","last_active":"2024-03-11T18:31:29.609646+0000","last_peered":"2024-03-11T18:31:29.609646+0000","last_clean":"2024-03-11T18:31:29.609646+0000","last_became_active":"2024-03-11T18:30:27.266394+0000","last_became_peered":"2024-03-11T18:30:27.266394+0000","last_unstale":"2024-03-11T18:31:29.609646+0000","last_undegraded":"2024-03-11T18:31:29.609646+0000","last_fullsized":"2024-03-11T18:31:29.609646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:01:24.115722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579253+0000","last_change":"2024-03-11T18:30:28.039476+0000","last_active":"2024-03-11T18:31:29.579253+0000","last_peered":"2024-03-11T18:31:29.579253+0000","last_clean":"2024-03-11T18:31:29.579253+0000","last_became_active":"2024-03-11T18:30:28.038897+0000","last_became_peered":"2024-03-11T18:30:28.038897+0000","last_unstale":"2024-03-11T18:31:29.579253+0000","last_undegraded":"2024-03-11T18:31:29.579253+0000","last_fullsized":"2024-03-11T18:31:29.579253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:36:16.103173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578819+0000","last_change":"2024-03-11T18:30:32.319516+0000","last_active":"2024-03-11T18:31:29.578819+0000","last_peered":"2024-03-11T18:31:29.578819+0000","last_clean":"2024-03-11T18:31:29.578819+0000","last_became_active":"2024-03-11T18:30:32.319117+0000","last_became_peered":"2024-03-11T18:30:32.319117+0000","last_unstale":"2024-03-11T18:31:29.578819+0000","last_undegraded":"2024-03-11T18:31:29.578819+0000","last_fullsized":"2024-03-11T18:31:29.578819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:39.797872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610559+0000","last_change":"2024-03-11T18:30:27.267543+0000","last_active":"2024-03-11T18:31:29.610559+0000","last_peered":"2024-03-11T18:31:29.610559+0000","last_clean":"2024-03-11T18:31:29.610559+0000","last_became_active":"2024-03-11T18:30:27.267235+0000","last_became_peered":"2024-03-11T18:30:27.267235+0000","last_unstale":"2024-03-11T18:31:29.610559+0000","last_undegraded":"2024-03-11T18:31:29.610559+0000","last_fullsized":"2024-03-11T18:31:29.610559+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:56:24.556896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586578+0000","last_change":"2024-03-11T18:30:28.039401+0000","last_active":"2024-03-11T18:31:29.586578+0000","last_peered":"2024-03-11T18:31:29.586578+0000","last_clean":"2024-03-11T18:31:29.586578+0000","last_became_active":"2024-03-11T18:30:28.039054+0000","last_became_peered":"2024-03-11T18:30:28.039054+0000","last_unstale":"2024-03-11T18:31:29.586578+0000","last_undegraded":"2024-03-11T18:31:29.586578+0000","last_fullsized":"2024-03-11T18:31:29.586578+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:01.701304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609860+0000","last_change":"2024-03-11T18:30:32.527930+0000","last_active":"2024-03-11T18:31:29.609860+0000","last_peered":"2024-03-11T18:31:29.609860+0000","last_clean":"2024-03-11T18:31:29.609860+0000","last_became_active":"2024-03-11T18:30:32.527371+0000","last_became_peered":"2024-03-11T18:30:32.527371+0000","last_unstale":"2024-03-11T18:31:29.609860+0000","last_undegraded":"2024-03-11T18:31:29.609860+0000","last_fullsized":"2024-03-11T18:31:29.609860+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:09:52.361924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591575+0000","last_change":"2024-03-11T18:30:27.281098+0000","last_active":"2024-03-11T18:31:29.591575+0000","last_peered":"2024-03-11T18:31:29.591575+0000","last_clean":"2024-03-11T18:31:29.591575+0000","last_became_active":"2024-03-11T18:30:27.280796+0000","last_became_peered":"2024-03-11T18:30:27.280796+0000","last_unstale":"2024-03-11T18:31:29.591575+0000","last_undegraded":"2024-03-11T18:31:29.591575+0000","last_fullsized":"2024-03-11T18:31:29.591575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:58:28.661626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609825+0000","last_change":"2024-03-11T18:30:28.054079+0000","last_active":"2024-03-11T18:31:29.609825+0000","last_peered":"2024-03-11T18:31:29.609825+0000","last_clean":"2024-03-11T18:31:29.609825+0000","last_became_active":"2024-03-11T18:30:28.053715+0000","last_became_peered":"2024-03-11T18:30:28.053715+0000","last_unstale":"2024-03-11T18:31:29.609825+0000","last_undegraded":"2024-03-11T18:31:29.609825+0000","last_fullsized":"2024-03-11T18:31:29.609825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:05.211602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581769+0000","last_change":"2024-03-11T18:30:32.311132+0000","last_active":"2024-03-11T18:31:29.581769+0000","last_peered":"2024-03-11T18:31:29.581769+0000","last_clean":"2024-03-11T18:31:29.581769+0000","last_became_active":"2024-03-11T18:30:32.309585+0000","last_became_peered":"2024-03-11T18:30:32.309585+0000","last_unstale":"2024-03-11T18:31:29.581769+0000","last_undegraded":"2024-03-11T18:31:29.581769+0000","last_fullsized":"2024-03-11T18:31:29.581769+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:57.272479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584220+0000","last_change":"2024-03-11T18:30:27.288080+0000","last_active":"2024-03-11T18:31:29.584220+0000","last_peered":"2024-03-11T18:31:29.584220+0000","last_clean":"2024-03-11T18:31:29.584220+0000","last_became_active":"2024-03-11T18:30:27.287776+0000","last_became_peered":"2024-03-11T18:30:27.287776+0000","last_unstale":"2024-03-11T18:31:29.584220+0000","last_undegraded":"2024-03-11T18:31:29.584220+0000","last_fullsized":"2024-03-11T18:31:29.584220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:31:22.081753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609889+0000","last_change":"2024-03-11T18:30:28.062011+0000","last_active":"2024-03-11T18:31:29.609889+0000","last_peered":"2024-03-11T18:31:29.609889+0000","last_clean":"2024-03-11T18:31:29.609889+0000","last_became_active":"2024-03-11T18:30:28.061592+0000","last_became_peered":"2024-03-11T18:30:28.061592+0000","last_unstale":"2024-03-11T18:31:29.609889+0000","last_undegraded":"2024-03-11T18:31:29.609889+0000","last_fullsized":"2024-03-11T18:31:29.609889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:01:10.013980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581357+0000","last_change":"2024-03-11T18:30:32.528123+0000","last_active":"2024-03-11T18:31:29.581357+0000","last_peered":"2024-03-11T18:31:29.581357+0000","last_clean":"2024-03-11T18:31:29.581357+0000","last_became_active":"2024-03-11T18:30:32.527611+0000","last_became_peered":"2024-03-11T18:30:32.527611+0000","last_unstale":"2024-03-11T18:31:29.581357+0000","last_undegraded":"2024-03-11T18:31:29.581357+0000","last_fullsized":"2024-03-11T18:31:29.581357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:31:28.501667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578964+0000","last_change":"2024-03-11T18:30:28.036930+0000","last_active":"2024-03-11T18:31:29.578964+0000","last_peered":"2024-03-11T18:31:29.578964+0000","last_clean":"2024-03-11T18:31:29.578964+0000","last_became_active":"2024-03-11T18:30:28.036286+0000","last_became_peered":"2024-03-11T18:30:28.036286+0000","last_unstale":"2024-03-11T18:31:29.578964+0000","last_undegraded":"2024-03-11T18:31:29.578964+0000","last_fullsized":"2024-03-11T18:31:29.578964+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:10:40.239284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":116,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:32:24.627762+0000","last_change":"2024-03-11T18:30:27.287838+0000","last_active":"2024-03-11T18:32:24.627762+0000","last_peered":"2024-03-11T18:32:24.627762+0000","last_clean":"2024-03-11T18:32:24.627762+0000","last_became_active":"2024-03-11T18:30:27.287447+0000","last_became_peered":"2024-03-11T18:30:27.287447+0000","last_unstale":"2024-03-11T18:32:24.627762+0000","last_undegraded":"2024-03-11T18:32:24.627762+0000","last_fullsized":"2024-03-11T18:32:24.627762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:47.242977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.617874+0000","last_change":"2024-03-11T18:31:29.617874+0000","last_active":"2024-03-11T18:31:29.617874+0000","last_peered":"2024-03-11T18:31:29.617874+0000","last_clean":"2024-03-11T18:31:29.617874+0000","last_became_active":"2024-03-11T18:31:29.610902+0000","last_became_peered":"2024-03-11T18:31:29.610902+0000","last_unstale":"2024-03-11T18:31:29.617874+0000","last_undegraded":"2024-03-11T18:31:29.617874+0000","last_fullsized":"2024-03-11T18:31:29.617874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:29:23.078025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610215+0000","last_change":"2024-03-11T18:30:28.037906+0000","last_active":"2024-03-11T18:31:29.610215+0000","last_peered":"2024-03-11T18:31:29.610215+0000","last_clean":"2024-03-11T18:31:29.610215+0000","last_became_active":"2024-03-11T18:30:28.034143+0000","last_became_peered":"2024-03-11T18:30:28.034143+0000","last_unstale":"2024-03-11T18:31:29.610215+0000","last_undegraded":"2024-03-11T18:31:29.610215+0000","last_fullsized":"2024-03-11T18:31:29.610215+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:13:36.728143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585543+0000","last_change":"2024-03-11T18:30:27.273520+0000","last_active":"2024-03-11T18:31:29.585543+0000","last_peered":"2024-03-11T18:31:29.585543+0000","last_clean":"2024-03-11T18:31:29.585543+0000","last_became_active":"2024-03-11T18:30:27.273141+0000","last_became_peered":"2024-03-11T18:30:27.273141+0000","last_unstale":"2024-03-11T18:31:29.585543+0000","last_undegraded":"2024-03-11T18:31:29.585543+0000","last_fullsized":"2024-03-11T18:31:29.585543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:16:51.467579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":362,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583899+0000","last_change":"2024-03-11T18:29:08.539862+0000","last_active":"2024-03-11T18:31:29.583899+0000","last_peered":"2024-03-11T18:31:29.583899+0000","last_clean":"2024-03-11T18:31:29.583899+0000","last_became_active":"2024-03-11T18:29:08.539403+0000","last_became_peered":"2024-03-11T18:29:08.539403+0000","last_unstale":"2024-03-11T18:31:29.583899+0000","last_undegraded":"2024-03-11T18:31:29.583899+0000","last_fullsized":"2024-03-11T18:31:29.583899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:25:13.035017+0000","last_clean_scrub_stamp":"2024-03-11T18:25:13.035017+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:31:24.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579774+0000","last_change":"2024-03-11T18:30:32.528841+0000","last_active":"2024-03-11T18:31:29.579774+0000","last_peered":"2024-03-11T18:31:29.579774+0000","last_clean":"2024-03-11T18:31:29.579774+0000","last_became_active":"2024-03-11T18:30:32.528320+0000","last_became_peered":"2024-03-11T18:30:32.528320+0000","last_unstale":"2024-03-11T18:31:29.579774+0000","last_undegraded":"2024-03-11T18:31:29.579774+0000","last_fullsized":"2024-03-11T18:31:29.579774+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:58:26.354490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580410+0000","last_change":"2024-03-11T18:30:28.047552+0000","last_active":"2024-03-11T18:31:29.580410+0000","last_peered":"2024-03-11T18:31:29.580410+0000","last_clean":"2024-03-11T18:31:29.580410+0000","last_became_active":"2024-03-11T18:30:28.046470+0000","last_became_peered":"2024-03-11T18:30:28.046470+0000","last_unstale":"2024-03-11T18:31:29.580410+0000","last_undegraded":"2024-03-11T18:31:29.580410+0000","last_fullsized":"2024-03-11T18:31:29.580410+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:58:23.080406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584138+0000","last_change":"2024-03-11T18:30:27.269416+0000","last_active":"2024-03-11T18:31:29.584138+0000","last_peered":"2024-03-11T18:31:29.584138+0000","last_clean":"2024-03-11T18:31:29.584138+0000","last_became_active":"2024-03-11T18:30:27.268951+0000","last_became_peered":"2024-03-11T18:30:27.268951+0000","last_unstale":"2024-03-11T18:31:29.584138+0000","last_undegraded":"2024-03-11T18:31:29.584138+0000","last_fullsized":"2024-03-11T18:31:29.584138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:03:16.138585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586398+0000","last_change":"2024-03-11T18:30:32.309247+0000","last_active":"2024-03-11T18:31:29.586398+0000","last_peered":"2024-03-11T18:31:29.586398+0000","last_clean":"2024-03-11T18:31:29.586398+0000","last_became_active":"2024-03-11T18:30:32.308761+0000","last_became_peered":"2024-03-11T18:30:32.308761+0000","last_unstale":"2024-03-11T18:31:29.586398+0000","last_undegraded":"2024-03-11T18:31:29.586398+0000","last_fullsized":"2024-03-11T18:31:29.586398+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:40:10.525154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610171+0000","last_change":"2024-03-11T18:30:27.279543+0000","last_active":"2024-03-11T18:31:29.610171+0000","last_peered":"2024-03-11T18:31:29.610171+0000","last_clean":"2024-03-11T18:31:29.610171+0000","last_became_active":"2024-03-11T18:30:27.279203+0000","last_became_peered":"2024-03-11T18:30:27.279203+0000","last_unstale":"2024-03-11T18:31:29.610171+0000","last_undegraded":"2024-03-11T18:31:29.610171+0000","last_fullsized":"2024-03-11T18:31:29.610171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:31.792757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581955+0000","last_change":"2024-03-11T18:30:28.060508+0000","last_active":"2024-03-11T18:31:29.581955+0000","last_peered":"2024-03-11T18:31:29.581955+0000","last_clean":"2024-03-11T18:31:29.581955+0000","last_became_active":"2024-03-11T18:30:28.059074+0000","last_became_peered":"2024-03-11T18:30:28.059074+0000","last_unstale":"2024-03-11T18:31:29.581955+0000","last_undegraded":"2024-03-11T18:31:29.581955+0000","last_fullsized":"2024-03-11T18:31:29.581955+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:34.204704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609508+0000","last_change":"2024-03-11T18:30:32.528017+0000","last_active":"2024-03-11T18:31:29.609508+0000","last_peered":"2024-03-11T18:31:29.609508+0000","last_clean":"2024-03-11T18:31:29.609508+0000","last_became_active":"2024-03-11T18:30:32.527518+0000","last_became_peered":"2024-03-11T18:30:32.527518+0000","last_unstale":"2024-03-11T18:31:29.609508+0000","last_undegraded":"2024-03-11T18:31:29.609508+0000","last_fullsized":"2024-03-11T18:31:29.609508+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:13:09.095889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594176+0000","last_change":"2024-03-11T18:30:27.275540+0000","last_active":"2024-03-11T18:31:29.594176+0000","last_peered":"2024-03-11T18:31:29.594176+0000","last_clean":"2024-03-11T18:31:29.594176+0000","last_became_active":"2024-03-11T18:30:27.275070+0000","last_became_peered":"2024-03-11T18:30:27.275070+0000","last_unstale":"2024-03-11T18:31:29.594176+0000","last_undegraded":"2024-03-11T18:31:29.594176+0000","last_fullsized":"2024-03-11T18:31:29.594176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:17:59.729542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610628+0000","last_change":"2024-03-11T18:30:28.045096+0000","last_active":"2024-03-11T18:31:29.610628+0000","last_peered":"2024-03-11T18:31:29.610628+0000","last_clean":"2024-03-11T18:31:29.610628+0000","last_became_active":"2024-03-11T18:30:28.044408+0000","last_became_peered":"2024-03-11T18:30:28.044408+0000","last_unstale":"2024-03-11T18:31:29.610628+0000","last_undegraded":"2024-03-11T18:31:29.610628+0000","last_fullsized":"2024-03-11T18:31:29.610628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:49:59.426247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580539+0000","last_change":"2024-03-11T18:30:32.303288+0000","last_active":"2024-03-11T18:31:29.580539+0000","last_peered":"2024-03-11T18:31:29.580539+0000","last_clean":"2024-03-11T18:31:29.580539+0000","last_became_active":"2024-03-11T18:30:32.302746+0000","last_became_peered":"2024-03-11T18:30:32.302746+0000","last_unstale":"2024-03-11T18:31:29.580539+0000","last_undegraded":"2024-03-11T18:31:29.580539+0000","last_fullsized":"2024-03-11T18:31:29.580539+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:04:24.541448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584867+0000","last_change":"2024-03-11T18:30:27.272880+0000","last_active":"2024-03-11T18:31:29.584867+0000","last_peered":"2024-03-11T18:31:29.584867+0000","last_clean":"2024-03-11T18:31:29.584867+0000","last_became_active":"2024-03-11T18:30:27.272611+0000","last_became_peered":"2024-03-11T18:30:27.272611+0000","last_unstale":"2024-03-11T18:31:29.584867+0000","last_undegraded":"2024-03-11T18:31:29.584867+0000","last_fullsized":"2024-03-11T18:31:29.584867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:41:10.322990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.591866+0000","last_change":"2024-03-11T18:30:28.065613+0000","last_active":"2024-03-11T18:31:29.591866+0000","last_peered":"2024-03-11T18:31:29.591866+0000","last_clean":"2024-03-11T18:31:29.591866+0000","last_became_active":"2024-03-11T18:30:28.060520+0000","last_became_peered":"2024-03-11T18:30:28.060520+0000","last_unstale":"2024-03-11T18:31:29.591866+0000","last_undegraded":"2024-03-11T18:31:29.591866+0000","last_fullsized":"2024-03-11T18:31:29.591866+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:09:44.530168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.611177+0000","last_change":"2024-03-11T18:31:29.611177+0000","last_active":"2024-03-11T18:31:29.611177+0000","last_peered":"2024-03-11T18:31:29.611177+0000","last_clean":"2024-03-11T18:31:29.611177+0000","last_became_active":"2024-03-11T18:31:29.610717+0000","last_became_peered":"2024-03-11T18:31:29.610717+0000","last_unstale":"2024-03-11T18:31:29.611177+0000","last_undegraded":"2024-03-11T18:31:29.611177+0000","last_fullsized":"2024-03-11T18:31:29.611177+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:41.226064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,8],"acting":[4,5,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608868+0000","last_change":"2024-03-11T18:30:27.261435+0000","last_active":"2024-03-11T18:31:29.608868+0000","last_peered":"2024-03-11T18:31:29.608868+0000","last_clean":"2024-03-11T18:31:29.608868+0000","last_became_active":"2024-03-11T18:30:27.260070+0000","last_became_peered":"2024-03-11T18:30:27.260070+0000","last_unstale":"2024-03-11T18:31:29.608868+0000","last_undegraded":"2024-03-11T18:31:29.608868+0000","last_fullsized":"2024-03-11T18:31:29.608868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:11.785818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":26,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609820+0000","last_change":"2024-03-11T18:30:28.041708+0000","last_active":"2024-03-11T18:31:29.609820+0000","last_peered":"2024-03-11T18:31:29.609820+0000","last_clean":"2024-03-11T18:31:29.609820+0000","last_became_active":"2024-03-11T18:30:28.039647+0000","last_became_peered":"2024-03-11T18:30:28.039647+0000","last_unstale":"2024-03-11T18:31:29.609820+0000","last_undegraded":"2024-03-11T18:31:29.609820+0000","last_fullsized":"2024-03-11T18:31:29.609820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:01:00.624175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.596205+0000","last_change":"2024-03-11T18:30:32.326512+0000","last_active":"2024-03-11T18:31:29.596205+0000","last_peered":"2024-03-11T18:31:29.596205+0000","last_clean":"2024-03-11T18:31:29.596205+0000","last_became_active":"2024-03-11T18:30:32.325253+0000","last_became_peered":"2024-03-11T18:30:32.325253+0000","last_unstale":"2024-03-11T18:31:29.596205+0000","last_undegraded":"2024-03-11T18:31:29.596205+0000","last_fullsized":"2024-03-11T18:31:29.596205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:48.589769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609610+0000","last_change":"2024-03-11T18:30:27.286194+0000","last_active":"2024-03-11T18:31:29.609610+0000","last_peered":"2024-03-11T18:31:29.609610+0000","last_clean":"2024-03-11T18:31:29.609610+0000","last_became_active":"2024-03-11T18:30:27.284877+0000","last_became_peered":"2024-03-11T18:30:27.284877+0000","last_unstale":"2024-03-11T18:31:29.609610+0000","last_undegraded":"2024-03-11T18:31:29.609610+0000","last_fullsized":"2024-03-11T18:31:29.609610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:56:20.324266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586216+0000","last_change":"2024-03-11T18:30:28.061038+0000","last_active":"2024-03-11T18:31:29.586216+0000","last_peered":"2024-03-11T18:31:29.586216+0000","last_clean":"2024-03-11T18:31:29.586216+0000","last_became_active":"2024-03-11T18:30:28.060701+0000","last_became_peered":"2024-03-11T18:30:28.060701+0000","last_unstale":"2024-03-11T18:31:29.586216+0000","last_undegraded":"2024-03-11T18:31:29.586216+0000","last_fullsized":"2024-03-11T18:31:29.586216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:40:11.838309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585027+0000","last_change":"2024-03-11T18:30:32.316762+0000","last_active":"2024-03-11T18:31:29.585027+0000","last_peered":"2024-03-11T18:31:29.585027+0000","last_clean":"2024-03-11T18:31:29.585027+0000","last_became_active":"2024-03-11T18:30:32.316318+0000","last_became_peered":"2024-03-11T18:30:32.316318+0000","last_unstale":"2024-03-11T18:31:29.585027+0000","last_undegraded":"2024-03-11T18:31:29.585027+0000","last_fullsized":"2024-03-11T18:31:29.585027+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:50:29.174005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580546+0000","last_change":"2024-03-11T18:30:27.276636+0000","last_active":"2024-03-11T18:31:29.580546+0000","last_peered":"2024-03-11T18:31:29.580546+0000","last_clean":"2024-03-11T18:31:29.580546+0000","last_became_active":"2024-03-11T18:30:27.276290+0000","last_became_peered":"2024-03-11T18:30:27.276290+0000","last_unstale":"2024-03-11T18:31:29.580546+0000","last_undegraded":"2024-03-11T18:31:29.580546+0000","last_fullsized":"2024-03-11T18:31:29.580546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:27:34.272568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610416+0000","last_change":"2024-03-11T18:30:28.067822+0000","last_active":"2024-03-11T18:31:29.610416+0000","last_peered":"2024-03-11T18:31:29.610416+0000","last_clean":"2024-03-11T18:31:29.610416+0000","last_became_active":"2024-03-11T18:30:28.067379+0000","last_became_peered":"2024-03-11T18:30:28.067379+0000","last_unstale":"2024-03-11T18:31:29.610416+0000","last_undegraded":"2024-03-11T18:31:29.610416+0000","last_fullsized":"2024-03-11T18:31:29.610416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:27.730186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584814+0000","last_change":"2024-03-11T18:30:32.527845+0000","last_active":"2024-03-11T18:31:29.584814+0000","last_peered":"2024-03-11T18:31:29.584814+0000","last_clean":"2024-03-11T18:31:29.584814+0000","last_became_active":"2024-03-11T18:30:32.527356+0000","last_became_peered":"2024-03-11T18:30:32.527356+0000","last_unstale":"2024-03-11T18:31:29.584814+0000","last_undegraded":"2024-03-11T18:31:29.584814+0000","last_fullsized":"2024-03-11T18:31:29.584814+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:44:32.005118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610309+0000","last_change":"2024-03-11T18:30:27.267263+0000","last_active":"2024-03-11T18:31:29.610309+0000","last_peered":"2024-03-11T18:31:29.610309+0000","last_clean":"2024-03-11T18:31:29.610309+0000","last_became_active":"2024-03-11T18:30:27.266893+0000","last_became_peered":"2024-03-11T18:30:27.266893+0000","last_unstale":"2024-03-11T18:31:29.610309+0000","last_undegraded":"2024-03-11T18:31:29.610309+0000","last_fullsized":"2024-03-11T18:31:29.610309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:52:04.305623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584121+0000","last_change":"2024-03-11T18:30:28.054211+0000","last_active":"2024-03-11T18:31:29.584121+0000","last_peered":"2024-03-11T18:31:29.584121+0000","last_clean":"2024-03-11T18:31:29.584121+0000","last_became_active":"2024-03-11T18:30:28.053472+0000","last_became_peered":"2024-03-11T18:30:28.053472+0000","last_unstale":"2024-03-11T18:31:29.584121+0000","last_undegraded":"2024-03-11T18:31:29.584121+0000","last_fullsized":"2024-03-11T18:31:29.584121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:24:37.415224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.608936+0000","last_change":"2024-03-11T18:30:32.304049+0000","last_active":"2024-03-11T18:31:29.608936+0000","last_peered":"2024-03-11T18:31:29.608936+0000","last_clean":"2024-03-11T18:31:29.608936+0000","last_became_active":"2024-03-11T18:30:32.303410+0000","last_became_peered":"2024-03-11T18:30:32.303410+0000","last_unstale":"2024-03-11T18:31:29.608936+0000","last_undegraded":"2024-03-11T18:31:29.608936+0000","last_fullsized":"2024-03-11T18:31:29.608936+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:15.574274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581052+0000","last_change":"2024-03-11T18:30:27.263269+0000","last_active":"2024-03-11T18:31:29.581052+0000","last_peered":"2024-03-11T18:31:29.581052+0000","last_clean":"2024-03-11T18:31:29.581052+0000","last_became_active":"2024-03-11T18:30:27.262892+0000","last_became_peered":"2024-03-11T18:30:27.262892+0000","last_unstale":"2024-03-11T18:31:29.581052+0000","last_undegraded":"2024-03-11T18:31:29.581052+0000","last_fullsized":"2024-03-11T18:31:29.581052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:59.753276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.584750+0000","last_change":"2024-03-11T18:30:28.043362+0000","last_active":"2024-03-11T18:31:29.584750+0000","last_peered":"2024-03-11T18:31:29.584750+0000","last_clean":"2024-03-11T18:31:29.584750+0000","last_became_active":"2024-03-11T18:30:28.042566+0000","last_became_peered":"2024-03-11T18:30:28.042566+0000","last_unstale":"2024-03-11T18:31:29.584750+0000","last_undegraded":"2024-03-11T18:31:29.584750+0000","last_fullsized":"2024-03-11T18:31:29.584750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:47:41.226647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.612747+0000","last_change":"2024-03-11T18:31:29.612747+0000","last_active":"2024-03-11T18:31:29.612747+0000","last_peered":"2024-03-11T18:31:29.612747+0000","last_clean":"2024-03-11T18:31:29.612747+0000","last_became_active":"2024-03-11T18:31:29.611916+0000","last_became_peered":"2024-03-11T18:31:29.611916+0000","last_unstale":"2024-03-11T18:31:29.612747+0000","last_undegraded":"2024-03-11T18:31:29.612747+0000","last_fullsized":"2024-03-11T18:31:29.612747+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:37:16.349623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579479+0000","last_change":"2024-03-11T18:30:27.271415+0000","last_active":"2024-03-11T18:31:29.579479+0000","last_peered":"2024-03-11T18:31:29.579479+0000","last_clean":"2024-03-11T18:31:29.579479+0000","last_became_active":"2024-03-11T18:30:27.271041+0000","last_became_peered":"2024-03-11T18:30:27.271041+0000","last_unstale":"2024-03-11T18:31:29.579479+0000","last_undegraded":"2024-03-11T18:31:29.579479+0000","last_fullsized":"2024-03-11T18:31:29.579479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:32:06.967984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.593827+0000","last_change":"2024-03-11T18:31:29.593827+0000","last_active":"2024-03-11T18:31:29.593827+0000","last_peered":"2024-03-11T18:31:29.593827+0000","last_clean":"2024-03-11T18:31:29.593827+0000","last_became_active":"2024-03-11T18:31:29.593485+0000","last_became_peered":"2024-03-11T18:31:29.593485+0000","last_unstale":"2024-03-11T18:31:29.593827+0000","last_undegraded":"2024-03-11T18:31:29.593827+0000","last_fullsized":"2024-03-11T18:31:29.593827+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:24:27.782987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.578619+0000","last_change":"2024-03-11T18:30:32.304111+0000","last_active":"2024-03-11T18:31:29.578619+0000","last_peered":"2024-03-11T18:31:29.578619+0000","last_clean":"2024-03-11T18:31:29.578619+0000","last_became_active":"2024-03-11T18:30:32.303420+0000","last_became_peered":"2024-03-11T18:30:32.303420+0000","last_unstale":"2024-03-11T18:31:29.578619+0000","last_undegraded":"2024-03-11T18:31:29.578619+0000","last_fullsized":"2024-03-11T18:31:29.578619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:31.277067+0000","last_clean_scrub_stamp":"2024-03-11T18:30:31.277067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:51.801772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.582234+0000","last_change":"2024-03-11T18:30:27.288015+0000","last_active":"2024-03-11T18:31:29.582234+0000","last_peered":"2024-03-11T18:31:29.582234+0000","last_clean":"2024-03-11T18:31:29.582234+0000","last_became_active":"2024-03-11T18:30:27.287528+0000","last_became_peered":"2024-03-11T18:30:27.287528+0000","last_unstale":"2024-03-11T18:31:29.582234+0000","last_undegraded":"2024-03-11T18:31:29.582234+0000","last_fullsized":"2024-03-11T18:31:29.582234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:26.229830+0000","last_clean_scrub_stamp":"2024-03-11T18:30:26.229830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:56:33.612120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.583592+0000","last_change":"2024-03-11T18:30:28.041888+0000","last_active":"2024-03-11T18:31:29.583592+0000","last_peered":"2024-03-11T18:31:29.583592+0000","last_clean":"2024-03-11T18:31:29.583592+0000","last_became_active":"2024-03-11T18:30:28.041506+0000","last_became_peered":"2024-03-11T18:30:28.041506+0000","last_unstale":"2024-03-11T18:31:29.583592+0000","last_undegraded":"2024-03-11T18:31:29.583592+0000","last_fullsized":"2024-03-11T18:31:29.583592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:53:46.308525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586241+0000","last_change":"2024-03-11T18:30:28.037778+0000","last_active":"2024-03-11T18:31:29.586241+0000","last_peered":"2024-03-11T18:31:29.586241+0000","last_clean":"2024-03-11T18:31:29.586241+0000","last_became_active":"2024-03-11T18:30:28.037428+0000","last_became_peered":"2024-03-11T18:30:28.037428+0000","last_unstale":"2024-03-11T18:31:29.586241+0000","last_undegraded":"2024-03-11T18:31:29.586241+0000","last_fullsized":"2024-03-11T18:31:29.586241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T18:30:35.810425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.585187+0000","last_change":"2024-03-11T18:31:29.585187+0000","last_active":"2024-03-11T18:31:29.585187+0000","last_peered":"2024-03-11T18:31:29.585187+0000","last_clean":"2024-03-11T18:31:29.585187+0000","last_became_active":"2024-03-11T18:31:29.584654+0000","last_became_peered":"2024-03-11T18:31:29.584654+0000","last_unstale":"2024-03-11T18:31:29.585187+0000","last_undegraded":"2024-03-11T18:31:29.585187+0000","last_fullsized":"2024-03-11T18:31:29.585187+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:07.081275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.580499+0000","last_change":"2024-03-11T18:30:28.060947+0000","last_active":"2024-03-11T18:31:29.580499+0000","last_peered":"2024-03-11T18:31:29.580499+0000","last_clean":"2024-03-11T18:31:29.580499+0000","last_became_active":"2024-03-11T18:30:28.060600+0000","last_became_peered":"2024-03-11T18:30:28.060600+0000","last_unstale":"2024-03-11T18:31:29.580499+0000","last_undegraded":"2024-03-11T18:31:29.580499+0000","last_fullsized":"2024-03-11T18:31:29.580499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:04:31.964347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.594893+0000","last_change":"2024-03-11T18:30:28.062110+0000","last_active":"2024-03-11T18:31:29.594893+0000","last_peered":"2024-03-11T18:31:29.594893+0000","last_clean":"2024-03-11T18:31:29.594893+0000","last_became_active":"2024-03-11T18:30:28.061427+0000","last_became_peered":"2024-03-11T18:30:28.061427+0000","last_unstale":"2024-03-11T18:31:29.594893+0000","last_undegraded":"2024-03-11T18:31:29.594893+0000","last_fullsized":"2024-03-11T18:31:29.594893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T19:36:32.452394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.598122+0000","last_change":"2024-03-11T18:31:29.598122+0000","last_active":"2024-03-11T18:31:29.598122+0000","last_peered":"2024-03-11T18:31:29.598122+0000","last_clean":"2024-03-11T18:31:29.598122+0000","last_became_active":"2024-03-11T18:31:29.597369+0000","last_became_peered":"2024-03-11T18:31:29.597369+0000","last_unstale":"2024-03-11T18:31:29.598122+0000","last_undegraded":"2024-03-11T18:31:29.598122+0000","last_fullsized":"2024-03-11T18:31:29.598122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:46:12.579516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.587042+0000","last_change":"2024-03-11T18:30:28.041817+0000","last_active":"2024-03-11T18:31:29.587042+0000","last_peered":"2024-03-11T18:31:29.587042+0000","last_clean":"2024-03-11T18:31:29.587042+0000","last_became_active":"2024-03-11T18:30:28.041346+0000","last_became_peered":"2024-03-11T18:30:28.041346+0000","last_unstale":"2024-03-11T18:31:29.587042+0000","last_undegraded":"2024-03-11T18:31:29.587042+0000","last_fullsized":"2024-03-11T18:31:29.587042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:26:28.544602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581230+0000","last_change":"2024-03-11T18:30:28.040820+0000","last_active":"2024-03-11T18:31:29.581230+0000","last_peered":"2024-03-11T18:31:29.581230+0000","last_clean":"2024-03-11T18:31:29.581230+0000","last_became_active":"2024-03-11T18:30:28.040381+0000","last_became_peered":"2024-03-11T18:30:28.040381+0000","last_unstale":"2024-03-11T18:31:29.581230+0000","last_undegraded":"2024-03-11T18:31:29.581230+0000","last_fullsized":"2024-03-11T18:31:29.581230+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:33:37.120786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.610760+0000","last_change":"2024-03-11T18:30:28.052050+0000","last_active":"2024-03-11T18:31:29.610760+0000","last_peered":"2024-03-11T18:31:29.610760+0000","last_clean":"2024-03-11T18:31:29.610760+0000","last_became_active":"2024-03-11T18:30:28.051685+0000","last_became_peered":"2024-03-11T18:30:28.051685+0000","last_unstale":"2024-03-11T18:31:29.610760+0000","last_undegraded":"2024-03-11T18:31:29.610760+0000","last_fullsized":"2024-03-11T18:31:29.610760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:37:02.777603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.579082+0000","last_change":"2024-03-11T18:30:28.053993+0000","last_active":"2024-03-11T18:31:29.579082+0000","last_peered":"2024-03-11T18:31:29.579082+0000","last_clean":"2024-03-11T18:31:29.579082+0000","last_became_active":"2024-03-11T18:30:28.053678+0000","last_became_peered":"2024-03-11T18:30:28.053678+0000","last_unstale":"2024-03-11T18:31:29.579082+0000","last_undegraded":"2024-03-11T18:31:29.579082+0000","last_fullsized":"2024-03-11T18:31:29.579082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:48.801078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609622+0000","last_change":"2024-03-11T18:30:28.051357+0000","last_active":"2024-03-11T18:31:29.609622+0000","last_peered":"2024-03-11T18:31:29.609622+0000","last_clean":"2024-03-11T18:31:29.609622+0000","last_became_active":"2024-03-11T18:30:28.051046+0000","last_became_peered":"2024-03-11T18:30:28.051046+0000","last_unstale":"2024-03-11T18:31:29.609622+0000","last_undegraded":"2024-03-11T18:31:29.609622+0000","last_fullsized":"2024-03-11T18:31:29.609622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:06:11.994754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.609246+0000","last_change":"2024-03-11T18:30:28.051628+0000","last_active":"2024-03-11T18:31:29.609246+0000","last_peered":"2024-03-11T18:31:29.609246+0000","last_clean":"2024-03-11T18:31:29.609246+0000","last_became_active":"2024-03-11T18:30:28.051129+0000","last_became_peered":"2024-03-11T18:30:28.051129+0000","last_unstale":"2024-03-11T18:31:29.609246+0000","last_undegraded":"2024-03-11T18:31:29.609246+0000","last_fullsized":"2024-03-11T18:31:29.609246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:55:47.568747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.581147+0000","last_change":"2024-03-11T18:30:28.065792+0000","last_active":"2024-03-11T18:31:29.581147+0000","last_peered":"2024-03-11T18:31:29.581147+0000","last_clean":"2024-03-11T18:31:29.581147+0000","last_became_active":"2024-03-11T18:30:28.064902+0000","last_became_peered":"2024-03-11T18:30:28.064902+0000","last_unstale":"2024-03-11T18:31:29.581147+0000","last_undegraded":"2024-03-11T18:31:29.581147+0000","last_fullsized":"2024-03-11T18:31:29.581147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:18.019872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T18:31:29.586665+0000","last_change":"2024-03-11T18:30:28.041663+0000","last_active":"2024-03-11T18:31:29.586665+0000","last_peered":"2024-03-11T18:31:29.586665+0000","last_clean":"2024-03-11T18:31:29.586665+0000","last_became_active":"2024-03-11T18:30:28.041257+0000","last_became_peered":"2024-03-11T18:30:28.041257+0000","last_unstale":"2024-03-11T18:31:29.586665+0000","last_undegraded":"2024-03-11T18:31:29.586665+0000","last_fullsized":"2024-03-11T18:31:29.586665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T18:30:27.244348+0000","last_clean_scrub_stamp":"2024-03-11T18:30:27.244348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:18.216295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67748,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":527,"num_read_kb":337,"num_write":166,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":203244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318309,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":507441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579946,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28716,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716436,"statfs":{"total":95995035648,"available":95965630464,"internally_reserved":0,"allocated":1736704,"data_stored":1162552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":511773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":507597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266749,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1103897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430274,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":502853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":566119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":504241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":566847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117075,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1048576,"data_stored":503047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":508907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691912,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1094998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T18:32:30.614 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T18:32:30.614 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T18:32:30.614 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T18:32:30.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T18:32:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:31.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:30 smithi002 ceph-mon[31328]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:31.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:30 smithi002 ceph-mon[31328]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T18:32:31.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:30 smithi023 ceph-mon[36406]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:31.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:30 smithi023 ceph-mon[36406]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T18:32:31.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:30 smithi165 ceph-mon[36634]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T18:32:31.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:30 smithi165 ceph-mon[36634]: osdmap e95: 12 total, 12 up, 12 in 2024-03-11T18:32:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:31 smithi002 ceph-mon[31328]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-11T18:32:32.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:31 smithi002 ceph-mon[31328]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T18:32:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:31 smithi165 ceph-mon[36634]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-11T18:32:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:31 smithi165 ceph-mon[36634]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T18:32:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:31 smithi023 ceph-mon[36406]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-11T18:32:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:31 smithi023 ceph-mon[36406]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 -- 172.21.15.2:0/4166314719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f503010e390 msgr2=0x7f5030110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 --2- 172.21.15.2:0/4166314719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f503010e390 0x7f5030110800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f502800b0a0 tx=0x7f502800b3b0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 -- 172.21.15.2:0/4166314719 shutdown_connections 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 --2- 172.21.15.2:0/4166314719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f503010e390 0x7f5030110800 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 --2- 172.21.15.2:0/4166314719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50300740a0 0x7f5030074480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 --2- 172.21.15.2:0/4166314719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 0x7f503010de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.396+0000 7f5037000700 1 -- 172.21.15.2:0/4166314719 >> 172.21.15.2:0/4166314719 conn(0x7f50300fd640 msgr2=0x7f50300ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.397+0000 7f5037000700 1 -- 172.21.15.2:0/4166314719 shutdown_connections 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.397+0000 7f5037000700 1 -- 172.21.15.2:0/4166314719 wait complete. 2024-03-11T18:32:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.398+0000 7f5037000700 1 Processor -- start 2024-03-11T18:32:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 -- start start 2024-03-11T18:32:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 0x7f503010acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f503010e390 0x7f503010b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503010b9d0 con 0x7f503010e390 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503010bb40 con 0x7f50300740a0 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5037000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50301aacf0 con 0x7f50300749c0 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.399+0000 7f5035c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51442/0 (socket says 172.21.15.2:51442) 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 -- 172.21.15.2:0/701512195 learned_addr learned my addr 172.21.15.2:0/701512195 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035200700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 0x7f503010acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5036600700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f503010e390 0x7f503010b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 -- 172.21.15.2:0/701512195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 msgr2=0x7f503010acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 0x7f503010acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 -- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f503010e390 msgr2=0x7f503010b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f503010e390 0x7f503010b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 -- 172.21.15.2:0/701512195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028009d00 con 0x7f50300740a0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.400+0000 7f5035c00700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f502400e9a0 tx=0x7f502400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.401+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024009790 con 0x7f50300740a0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.401+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5024010430 con 0x7f50300740a0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.401+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50301aaf50 con 0x7f50300740a0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.401+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024018680 con 0x7f50300740a0 2024-03-11T18:32:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.401+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50301ab410 con 0x7f50300740a0 2024-03-11T18:32:33.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.402+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5030075750 con 0x7f50300740a0 2024-03-11T18:32:33.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.404+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f5024010910 con 0x7f50300740a0 2024-03-11T18:32:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.405+0000 7f502e000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 0x7f50180a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.405+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5024014070 con 0x7f50300740a0 2024-03-11T18:32:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.405+0000 7f5035200700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 0x7f50180a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.406+0000 7f5035200700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 0x7f50180a4000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f503010c480 tx=0x7f501c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.408+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f50240967a0 con 0x7f50300740a0 2024-03-11T18:32:33.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.684+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5030113e30 con 0x7f50300740a0 2024-03-11T18:32:33.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.685+0000 7f502e000700 1 -- 172.21.15.2:0/701512195 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+180 (secure 0 0 0) 0x7f50240105a0 con 0x7f50300740a0 2024-03-11T18:32:33.687 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:33.687 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false}},"mutes":[]} 2024-03-11T18:32:33.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 msgr2=0x7f50180a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:33.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 0x7f50180a4000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f503010c480 tx=0x7f501c009260 comp rx=0 tx=0).stop 2024-03-11T18:32:33.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 msgr2=0x7f503010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:33.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f502400e9a0 tx=0x7f502400ecb0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 shutdown_connections 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f503010e390 0x7f503010b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50300740a0 0x7f503010dc10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f50180a1b40 0x7f50180a4000 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 --2- 172.21.15.2:0/701512195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50300749c0 0x7f503010acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 >> 172.21.15.2:0/701512195 conn(0x7f50300fd640 msgr2=0x7f503010fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 shutdown_connections 2024-03-11T18:32:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:33.689+0000 7f5037000700 1 -- 172.21.15.2:0/701512195 wait complete. 2024-03-11T18:32:34.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:33 smithi002 ceph-mon[31328]: pgmap v387: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:34.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:33 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/701512195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:33 smithi165 ceph-mon[36634]: pgmap v387: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:33 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/701512195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:34.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:33 smithi023 ceph-mon[36406]: pgmap v387: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:34.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:33 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/701512195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:36.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:35 smithi002 ceph-mon[31328]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 895 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:35 smithi165 ceph-mon[36634]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 895 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:36.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:35 smithi023 ceph-mon[36406]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 895 B/s wr, 5 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T18:32:37.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:36 smithi002 ceph-mon[31328]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T18:32:37.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:36 smithi002 ceph-mon[31328]: Cluster is now healthy 2024-03-11T18:32:37.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:36 smithi165 ceph-mon[36634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T18:32:37.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:36 smithi165 ceph-mon[36634]: Cluster is now healthy 2024-03-11T18:32:37.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:36 smithi023 ceph-mon[36406]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-11T18:32:37.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:36 smithi023 ceph-mon[36406]: Cluster is now healthy 2024-03-11T18:32:37.497 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T18:32:37.995 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:38.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:37 smithi002 ceph-mon[31328]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 895 B/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-03-11T18:32:38.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:37 smithi023 ceph-mon[36406]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 895 B/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-03-11T18:32:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:37 smithi165 ceph-mon[36634]: pgmap v389: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 895 B/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-03-11T18:32:40.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:39 smithi002 ceph-mon[31328]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 766 B/s wr, 7 op/s; 74 B/s, 1 objects/s recovering 2024-03-11T18:32:40.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 -- 172.21.15.2:0/3147346374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd060105e10 msgr2=0x7fd0601061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 --2- 172.21.15.2:0/3147346374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd060105e10 0x7fd0601061f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd0500098f0 tx=0x7fd050009c00 comp rx=0 tx=0).stop 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 -- 172.21.15.2:0/3147346374 shutdown_connections 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 --2- 172.21.15.2:0/3147346374 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd060105e10 0x7fd0601061f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 --2- 172.21.15.2:0/3147346374 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06010cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 --2- 172.21.15.2:0/3147346374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0600fffb0 0x7fd060100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 -- 172.21.15.2:0/3147346374 >> 172.21.15.2:0/3147346374 conn(0x7fd060075930 msgr2=0x7fd060075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 -- 172.21.15.2:0/3147346374 shutdown_connections 2024-03-11T18:32:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.228+0000 7fd064e00700 1 -- 172.21.15.2:0/3147346374 wait complete. 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.229+0000 7fd064e00700 1 Processor -- start 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 -- start start 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 0x7fd06019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd060105e10 0x7fd06019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06019f260 con 0x7fd060100970 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06019f3d0 con 0x7fd060105e10 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd064e00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06019f590 con 0x7fd0600fffb0 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd057e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd057e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53132/0 (socket says 172.21.15.2:53132) 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd057e00700 1 -- 172.21.15.2:0/2136205783 learned_addr learned my addr 172.21.15.2:0/2136205783 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.230+0000 7fd05f400700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 0x7fd06019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd05fe00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd060105e10 0x7fd06019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 -- 172.21.15.2:0/2136205783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 msgr2=0x7fd06019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 0x7fd06019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 -- 172.21.15.2:0/2136205783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd060105e10 msgr2=0x7fd06019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd060105e10 0x7fd06019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 -- 172.21.15.2:0/2136205783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058009d00 con 0x7fd060100970 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd05f400700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 0x7fd06019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:40.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.231+0000 7fd057e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd0480097b0 tx=0x7fd048009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.232+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048014820 con 0x7fd060100970 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.232+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd048010040 con 0x7fd060100970 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.232+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd050009520 con 0x7fd060100970 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.232+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048013d90 con 0x7fd060100970 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.232+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0601a6f00 con 0x7fd060100970 2024-03-11T18:32:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.233+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06010c700 con 0x7fd060100970 2024-03-11T18:32:40.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.235+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd048013490 con 0x7fd060100970 2024-03-11T18:32:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.236+0000 7fd05cc00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 0x7fd0400a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.237+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd0480cd3a0 con 0x7fd060100970 2024-03-11T18:32:40.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.237+0000 7fd05f400700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 0x7fd0400a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.237+0000 7fd05f400700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 0x7fd0400a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd050009810 tx=0x7fd050019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:40.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.241+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd048096190 con 0x7fd060100970 2024-03-11T18:32:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:39 smithi023 ceph-mon[36406]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 766 B/s wr, 7 op/s; 74 B/s, 1 objects/s recovering 2024-03-11T18:32:40.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:39 smithi165 ceph-mon[36634]: pgmap v390: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 766 B/s wr, 7 op/s; 74 B/s, 1 objects/s recovering 2024-03-11T18:32:40.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.522+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd06019f750 con 0x7fd060100970 2024-03-11T18:32:40.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.523+0000 7fd05cc00700 1 -- 172.21.15.2:0/2136205783 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fd048099850 con 0x7fd060100970 2024-03-11T18:32:40.525 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:40.525 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T18:32:40.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 msgr2=0x7fd0400a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 0x7fd0400a3f80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd050009810 tx=0x7fd050019040 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 msgr2=0x7fd06019a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd0480097b0 tx=0x7fd048009ac0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 shutdown_connections 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd0400a1ac0 0x7fd0400a3f80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0600fffb0 0x7fd06019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd060105e10 0x7fd06019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 --2- 172.21.15.2:0/2136205783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd060100970 0x7fd06019a8c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 >> 172.21.15.2:0/2136205783 conn(0x7fd060075930 msgr2=0x7fd0601040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:40.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 shutdown_connections 2024-03-11T18:32:40.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:40.526+0000 7fd064e00700 1 -- 172.21.15.2:0/2136205783 wait complete. 2024-03-11T18:32:41.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:40 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2136205783' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:41.325 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T18:32:41.326 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:32:41.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:40 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2136205783' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:40 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2136205783' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 -- 172.21.15.2:0/427018145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c108890 msgr2=0x7efc5c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 --2- 172.21.15.2:0/427018145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c108890 0x7efc5c108d10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efc44009900 tx=0x7efc4402bc40 comp rx=0 tx=0).stop 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 -- 172.21.15.2:0/427018145 shutdown_connections 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 --2- 172.21.15.2:0/427018145 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc5c109250 0x7efc5c10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 --2- 172.21.15.2:0/427018145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c108890 0x7efc5c108d10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 --2- 172.21.15.2:0/427018145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c1076a0 0x7efc5c107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 -- 172.21.15.2:0/427018145 >> 172.21.15.2:0/427018145 conn(0x7efc5c076640 msgr2=0x7efc5c078a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 -- 172.21.15.2:0/427018145 shutdown_connections 2024-03-11T18:32:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.563+0000 7efc5be00640 1 -- 172.21.15.2:0/427018145 wait complete. 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.564+0000 7efc5be00640 1 Processor -- start 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.564+0000 7efc5be00640 1 -- start start 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c108890 0x7efc5c19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc5c109250 0x7efc5c1a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1a5260 con 0x7efc5c108890 2024-03-11T18:32:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1a53d0 con 0x7efc5c1076a0 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5c1a5590 con 0x7efc5c109250 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56490/0 (socket says 172.21.15.2:56490) 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc5aa00640 1 -- 172.21.15.2:0/3177840292 learned_addr learned my addr 172.21.15.2:0/3177840292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.565+0000 7efc52000640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c108890 0x7efc5c19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 -- 172.21.15.2:0/3177840292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc5c109250 msgr2=0x7efc5c1a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc5c109250 0x7efc5c1a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 -- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c108890 msgr2=0x7efc5c19dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c108890 0x7efc5c19dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 -- 172.21.15.2:0/3177840292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44009560 con 0x7efc5c1076a0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5aa00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00d950 tx=0x7efc4c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c009c30 con 0x7efc5c1076a0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc4c009d90 con 0x7efc5c1076a0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c1a5870 con 0x7efc5c1076a0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00f880 con 0x7efc5c1076a0 2024-03-11T18:32:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.566+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5c1a5d50 con 0x7efc5c1076a0 2024-03-11T18:32:41.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.567+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5c10e8f0 con 0x7efc5c1076a0 2024-03-11T18:32:41.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.569+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7efc4c00f9e0 con 0x7efc5c1076a0 2024-03-11T18:32:41.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.571+0000 7efc53e00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 0x7efc3c0a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:41.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.571+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7efc4c0cda00 con 0x7efc5c1076a0 2024-03-11T18:32:41.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.571+0000 7efc52000640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 0x7efc3c0a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:41.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.572+0000 7efc52000640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 0x7efc3c0a3ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efc44000c00 tx=0x7efc4402c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:41.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.575+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc4c0967f0 con 0x7efc5c1076a0 2024-03-11T18:32:41.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.778+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc5c1130f0 con 0x7efc5c1076a0 2024-03-11T18:32:41.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.780+0000 7efc53e00640 1 -- 172.21.15.2:0/3177840292 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7efc4c099eb0 con 0x7efc5c1076a0 2024-03-11T18:32:41.782 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:41.783 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 msgr2=0x7efc3c0a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 0x7efc3c0a3ee0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efc44000c00 tx=0x7efc4402c7f0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 msgr2=0x7efc5c19d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00d950 tx=0x7efc4c00de10 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 shutdown_connections 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efc3c0a1a20 0x7efc3c0a3ee0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc5c109250 0x7efc5c1a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc5c108890 0x7efc5c19dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 --2- 172.21.15.2:0/3177840292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc5c1076a0 0x7efc5c19d770 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 >> 172.21.15.2:0/3177840292 conn(0x7efc5c076640 msgr2=0x7efc5c10bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 shutdown_connections 2024-03-11T18:32:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:41.783+0000 7efc5be00640 1 -- 172.21.15.2:0/3177840292 wait complete. 2024-03-11T18:32:41.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:32:41.804 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:32:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 -- 172.21.15.2:0/3662063911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c01023c0 msgr2=0x7f96c0102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 --2- 172.21.15.2:0/3662063911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c01023c0 0x7f96c0102840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009900 tx=0x7f96ac02bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 -- 172.21.15.2:0/3662063911 shutdown_connections 2024-03-11T18:32:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 --2- 172.21.15.2:0/3662063911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c0102d80 0x7f96c0109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 --2- 172.21.15.2:0/3662063911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c01023c0 0x7f96c0102840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 --2- 172.21.15.2:0/3662063911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 -- 172.21.15.2:0/3662063911 >> 172.21.15.2:0/3662063911 conn(0x7f96c00fc740 msgr2=0x7f96c00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 -- 172.21.15.2:0/3662063911 shutdown_connections 2024-03-11T18:32:42.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.025+0000 7f96c6200640 1 -- 172.21.15.2:0/3662063911 wait complete. 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.026+0000 7f96c6200640 1 Processor -- start 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 -- start start 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 0x7f96c01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c0102d80 0x7f96c01a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01a0fa0 con 0x7f96c0102d80 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01a1110 con 0x7f96c01023c0 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96c6200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c01a12d0 con 0x7f96c01011d0 2024-03-11T18:32:42.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96bf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96bea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 0x7f96c01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96bfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c0102d80 0x7f96c01a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96bf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57824/0 (socket says 172.21.15.2:57824) 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.027+0000 7f96bf400640 1 -- 172.21.15.2:0/698181108 learned_addr learned my addr 172.21.15.2:0/698181108 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 -- 172.21.15.2:0/698181108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 msgr2=0x7f96c01994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 -- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c0102d80 msgr2=0x7f96c01a0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c0102d80 0x7f96c01a0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 -- 172.21.15.2:0/698181108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac009560 con 0x7f96c01023c0 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bf400640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96bea00640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 0x7f96c01999f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f96ac000c00 tx=0x7f96ac005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.028+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac0043d0 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.029+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c01a1550 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.029+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96ac0052d0 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.029+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac003d40 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.029+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c01a1a80 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.030+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c010cbe0 con 0x7f96c01023c0 2024-03-11T18:32:42.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.032+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f96ac005440 con 0x7f96c01023c0 2024-03-11T18:32:42.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.033+0000 7f96b7e00640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 0x7f96a40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.033+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f96ac0efb10 con 0x7f96c01023c0 2024-03-11T18:32:42.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.033+0000 7f96bf400640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 0x7f96a40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.034+0000 7f96bf400640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 0x7f96a40a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96b0005f40 tx=0x7f96b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.035+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96ac0b8900 con 0x7f96c01023c0 2024-03-11T18:32:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.236+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96c010cda0 con 0x7f96c01023c0 2024-03-11T18:32:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.237+0000 7f96b7e00640 1 -- 172.21.15.2:0/698181108 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96ac0bbfc0 con 0x7f96c01023c0 2024-03-11T18:32:42.239 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:32:42.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 msgr2=0x7f96a40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 0x7f96a40a3ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96b0005f40 tx=0x7f96b0009500 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 msgr2=0x7f96c01999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 0x7f96c01999f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f96ac000c00 tx=0x7f96ac005e50 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 shutdown_connections 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f96a40a1a30 0x7f96a40a3ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96c0102d80 0x7f96c01a0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f96c01023c0 0x7f96c01999f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 --2- 172.21.15.2:0/698181108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96c01011d0 0x7f96c01994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 >> 172.21.15.2:0/698181108 conn(0x7f96c00fc740 msgr2=0x7f96c00733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 shutdown_connections 2024-03-11T18:32:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.240+0000 7f96c6200640 1 -- 172.21.15.2:0/698181108 wait complete. 2024-03-11T18:32:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:41 smithi002 ceph-mon[31328]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 296 B/s wr, 6 op/s; 67 B/s, 1 objects/s recovering 2024-03-11T18:32:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:41 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3177840292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:42.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:32:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:41 smithi165 ceph-mon[36634]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 296 B/s wr, 6 op/s; 67 B/s, 1 objects/s recovering 2024-03-11T18:32:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:41 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3177840292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:42.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:41 smithi023 ceph-mon[36406]: pgmap v391: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 296 B/s wr, 6 op/s; 67 B/s, 1 objects/s recovering 2024-03-11T18:32:42.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:41 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3177840292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:42.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 -- 172.21.15.2:0/3821882823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a81087d0 msgr2=0x7f58a8108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 --2- 172.21.15.2:0/3821882823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a81087d0 0x7f58a8108c50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5890009900 tx=0x7f589002bc40 comp rx=0 tx=0).stop 2024-03-11T18:32:42.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 -- 172.21.15.2:0/3821882823 shutdown_connections 2024-03-11T18:32:42.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 --2- 172.21.15.2:0/3821882823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a8109190 0x7f58a810fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 --2- 172.21.15.2:0/3821882823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a81087d0 0x7f58a8108c50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 --2- 172.21.15.2:0/3821882823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 0x7f58a8107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.493+0000 7f58a7200640 1 -- 172.21.15.2:0/3821882823 >> 172.21.15.2:0/3821882823 conn(0x7f58a8076570 msgr2=0x7f58a80789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.494+0000 7f58a7200640 1 -- 172.21.15.2:0/3821882823 shutdown_connections 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.494+0000 7f58a7200640 1 -- 172.21.15.2:0/3821882823 wait complete. 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.495+0000 7f58a7200640 1 Processor -- start 2024-03-11T18:32:42.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.495+0000 7f58a7200640 1 -- start start 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 0x7f58a819d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a8109190 0x7f58a81a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81a5180 con 0x7f58a81075e0 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81a52f0 con 0x7f58a81087d0 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81a54b0 con 0x7f58a8109190 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f589fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 0x7f58a819d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f589fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56510/0 (socket says 172.21.15.2:56510) 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f589fe00640 1 -- 172.21.15.2:0/1760368740 learned_addr learned my addr 172.21.15.2:0/1760368740 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f58a6800640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a8109190 0x7f58a81a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.496+0000 7f589fe00640 1 -- 172.21.15.2:0/1760368740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a8109190 msgr2=0x7f58a81a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589fe00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a8109190 0x7f58a81a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589fe00640 1 -- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 msgr2=0x7f58a819d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589fe00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 0x7f58a819d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589fe00640 1 -- 172.21.15.2:0/1760368740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0009840 con 0x7f58a81087d0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589fe00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5890006010 tx=0x7f5890005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58900043d0 con 0x7f58a81087d0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5890009560 con 0x7f58a81087d0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58900052d0 con 0x7f58a81087d0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5890003bc0 con 0x7f58a81087d0 2024-03-11T18:32:42.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.497+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a81a5ab0 con 0x7f58a81087d0 2024-03-11T18:32:42.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.498+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a810e920 con 0x7f58a81087d0 2024-03-11T18:32:42.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.501+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f5890003d40 con 0x7f58a81087d0 2024-03-11T18:32:42.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.502+0000 7f589ea00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 0x7f588c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.503+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f58900efa40 con 0x7f58a81087d0 2024-03-11T18:32:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.503+0000 7f58a5e00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 0x7f588c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.504+0000 7f58a5e00640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 0x7f588c0a3ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f58a8108630 tx=0x7f5898005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.507+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58900b8830 con 0x7f58a81087d0 2024-03-11T18:32:42.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.710+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f58a802cdd0 con 0x7f58a81087d0 2024-03-11T18:32:42.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.711+0000 7f589ea00640 1 -- 172.21.15.2:0/1760368740 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f58900bbef0 con 0x7f58a81087d0 2024-03-11T18:32:42.713 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 msgr2=0x7f588c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 0x7f588c0a3ef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f58a8108630 tx=0x7f5898005f90 comp rx=0 tx=0).stop 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 msgr2=0x7f58a819dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5890006010 tx=0x7f5890005e80 comp rx=0 tx=0).stop 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 shutdown_connections 2024-03-11T18:32:42.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f588c0a1a30 0x7f588c0a3ef0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58a8109190 0x7f58a81a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58a81087d0 0x7f58a819dc70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 --2- 172.21.15.2:0/1760368740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58a81075e0 0x7f58a819d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.714+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 >> 172.21.15.2:0/1760368740 conn(0x7f58a8076570 msgr2=0x7f58a810baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.715+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 shutdown_connections 2024-03-11T18:32:42.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.715+0000 7f58a7200640 1 -- 172.21.15.2:0/1760368740 wait complete. 2024-03-11T18:32:42.738 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:32:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 -- 172.21.15.2:0/1420383857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 msgr2=0x7fad3810b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 --2- 172.21.15.2:0/1420383857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad3810b620 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad30009ca0 tx=0x7fad3002c100 comp rx=0 tx=0).stop 2024-03-11T18:32:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 -- 172.21.15.2:0/1420383857 shutdown_connections 2024-03-11T18:32:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 --2- 172.21.15.2:0/1420383857 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad3810b620 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 --2- 172.21.15.2:0/1420383857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381043d0 0x7fad38104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 --2- 172.21.15.2:0/1420383857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381031e0 0x7fad38103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 -- 172.21.15.2:0/1420383857 >> 172.21.15.2:0/1420383857 conn(0x7fad380fe750 msgr2=0x7fad38100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 -- 172.21.15.2:0/1420383857 shutdown_connections 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.961+0000 7fad3ec00640 1 -- 172.21.15.2:0/1420383857 wait complete. 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 Processor -- start 2024-03-11T18:32:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 -- start start 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381043d0 0x7fad38199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad381a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381a0d10 con 0x7fad38104d90 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381a0e80 con 0x7fad381031e0 2024-03-11T18:32:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.963+0000 7fad3ec00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381a1040 con 0x7fad381043d0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad381a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad381a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53212/0 (socket says 172.21.15.2:53212) 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56524/0 (socket says 172.21.15.2:56524) 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3e200640 1 -- 172.21.15.2:0/2857515821 learned_addr learned my addr 172.21.15.2:0/2857515821 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3ce00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381043d0 0x7fad38199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 -- 172.21.15.2:0/2857515821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381043d0 msgr2=0x7fad38199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381043d0 0x7fad38199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 -- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 msgr2=0x7fad381a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad381a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.964+0000 7fad3d800640 1 -- 172.21.15.2:0/2857515821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24009490 con 0x7fad381031e0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad3d800640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00e9c0 tx=0x7fad2c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00ccc0 con 0x7fad381031e0 2024-03-11T18:32:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad30009900 con 0x7fad381031e0 2024-03-11T18:32:42.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad2c00ce20 con 0x7fad381031e0 2024-03-11T18:32:42.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c010430 con 0x7fad381031e0 2024-03-11T18:32:42.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.965+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad381a1650 con 0x7fad381031e0 2024-03-11T18:32:42.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.966+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3810a500 con 0x7fad381031e0 2024-03-11T18:32:42.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.968+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fad2c004000 con 0x7fad381031e0 2024-03-11T18:32:42.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.970+0000 7fad36000640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 0x7fad200a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:42.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.970+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fad2c014070 con 0x7fad381031e0 2024-03-11T18:32:42.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.970+0000 7fad3ce00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 0x7fad200a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:42.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.971+0000 7fad3ce00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 0x7fad200a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fad3819a730 tx=0x7fad24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:42.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:42.972+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad2c096a50 con 0x7fad381031e0 2024-03-11T18:32:43.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.179+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad3802ce40 con 0x7fad381031e0 2024-03-11T18:32:43.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.179+0000 7fad36000640 1 -- 172.21.15.2:0/2857515821 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad2c09a110 con 0x7fad381031e0 2024-03-11T18:32:43.181 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 msgr2=0x7fad200a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 0x7fad200a3ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fad3819a730 tx=0x7fad24005f90 comp rx=0 tx=0).stop 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 msgr2=0x7fad38199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00e9c0 tx=0x7fad2c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 shutdown_connections 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fad200a1a30 0x7fad200a3ef0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fad38104d90 0x7fad381a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad381043d0 0x7fad38199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 --2- 172.21.15.2:0/2857515821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad381031e0 0x7fad38199220 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 >> 172.21.15.2:0/2857515821 conn(0x7fad380fe750 msgr2=0x7fad381076d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.182+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 shutdown_connections 2024-03-11T18:32:43.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.183+0000 7fad3ec00640 1 -- 172.21.15.2:0/2857515821 wait complete. 2024-03-11T18:32:43.203 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:32:43.286 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:43 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/698181108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.286 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:43 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1760368740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:43 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/698181108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:43 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1760368740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:43 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/698181108' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:43 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1760368740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 -- 172.21.15.2:0/2735314328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff574109190 msgr2=0x7ff57410fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 --2- 172.21.15.2:0/2735314328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff574109190 0x7ff57410fa20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff56c009be0 tx=0x7ff56c02bef0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 -- 172.21.15.2:0/2735314328 shutdown_connections 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 --2- 172.21.15.2:0/2735314328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff574109190 0x7ff57410fa20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 --2- 172.21.15.2:0/2735314328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741087d0 0x7ff574108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 --2- 172.21.15.2:0/2735314328 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5741075e0 0x7ff574107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 -- 172.21.15.2:0/2735314328 >> 172.21.15.2:0/2735314328 conn(0x7ff574076570 msgr2=0x7ff5740789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 -- 172.21.15.2:0/2735314328 shutdown_connections 2024-03-11T18:32:43.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.431+0000 7ff579e00640 1 -- 172.21.15.2:0/2735314328 wait complete. 2024-03-11T18:32:43.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.432+0000 7ff579e00640 1 Processor -- start 2024-03-11T18:32:43.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.432+0000 7ff579e00640 1 -- start start 2024-03-11T18:32:43.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 0x7ff57419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 0x7ff5741a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741a50a0 con 0x7ff5741075e0 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741a5210 con 0x7ff5741087d0 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5741a53d0 con 0x7ff574109190 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff578a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff578a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53226/0 (socket says 172.21.15.2:53226) 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff578a00640 1 -- 172.21.15.2:0/881301726 learned_addr learned my addr 172.21.15.2:0/881301726 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff573e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 0x7ff57419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff579400640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 0x7ff5741a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.433+0000 7ff578a00640 1 -- 172.21.15.2:0/881301726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 msgr2=0x7ff5741a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff578a00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 0x7ff5741a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff578a00640 1 -- 172.21.15.2:0/881301726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 msgr2=0x7ff57419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff578a00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 0x7ff57419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff578a00640 1 -- 172.21.15.2:0/881301726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568009490 con 0x7ff5741075e0 2024-03-11T18:32:43.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff579400640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 0x7ff5741a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:43.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff573e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 0x7ff57419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:43.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff578a00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff56800e9c0 tx=0x7ff56800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:43.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56800cde0 con 0x7ff5741075e0 2024-03-11T18:32:43.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff568010910 con 0x7ff5741075e0 2024-03-11T18:32:43.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff560009560 con 0x7ff5741075e0 2024-03-11T18:32:43.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568018960 con 0x7ff5741075e0 2024-03-11T18:32:43.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.434+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff56c009840 con 0x7ff5741075e0 2024-03-11T18:32:43.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.435+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57410e830 con 0x7ff5741075e0 2024-03-11T18:32:43.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.438+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7ff568010a80 con 0x7ff5741075e0 2024-03-11T18:32:43.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.439+0000 7ff571600640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 0x7ff55c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.439+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ff568014070 con 0x7ff5741075e0 2024-03-11T18:32:43.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.439+0000 7ff573e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 0x7ff55c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.440+0000 7ff573e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 0x7ff55c0a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff57419eac0 tx=0x7ff560009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:43.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.443+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff568096bc0 con 0x7ff5741075e0 2024-03-11T18:32:43.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.642+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff57402ce40 con 0x7ff5741075e0 2024-03-11T18:32:43.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.643+0000 7ff571600640 1 -- 172.21.15.2:0/881301726 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff56809a280 con 0x7ff5741075e0 2024-03-11T18:32:43.645 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:32:43.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 msgr2=0x7ff55c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 0x7ff55c0a3ef0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff57419eac0 tx=0x7ff560009260 comp rx=0 tx=0).stop 2024-03-11T18:32:43.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 msgr2=0x7ff57419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff56800e9c0 tx=0x7ff56800ee80 comp rx=0 tx=0).stop 2024-03-11T18:32:43.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 shutdown_connections 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff55c0a1a30 0x7ff55c0a3ef0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff574109190 0x7ff5741a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff5741087d0 0x7ff57419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 --2- 172.21.15.2:0/881301726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff5741075e0 0x7ff57419d5b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.645+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 >> 172.21.15.2:0/881301726 conn(0x7ff574076570 msgr2=0x7ff57410ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.646+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 shutdown_connections 2024-03-11T18:32:43.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.646+0000 7ff579e00640 1 -- 172.21.15.2:0/881301726 wait complete. 2024-03-11T18:32:43.668 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:32:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 -- 172.21.15.2:0/2404591985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 msgr2=0x7f2110102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 --2- 172.21.15.2:0/2404591985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f2110102710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20f8009ca0 tx=0x7f20f802c0f0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 -- 172.21.15.2:0/2404591985 shutdown_connections 2024-03-11T18:32:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 --2- 172.21.15.2:0/2404591985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2110102c50 0x7f21101094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 --2- 172.21.15.2:0/2404591985 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f2110102710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 --2- 172.21.15.2:0/2404591985 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f21101010a0 0x7f21101014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 -- 172.21.15.2:0/2404591985 >> 172.21.15.2:0/2404591985 conn(0x7f21100fc610 msgr2=0x7f21100fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.886+0000 7f210f200640 1 -- 172.21.15.2:0/2404591985 shutdown_connections 2024-03-11T18:32:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.887+0000 7f210f200640 1 -- 172.21.15.2:0/2404591985 wait complete. 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.887+0000 7f210f200640 1 Processor -- start 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 -- start start 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2110102c50 0x7f21101a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21101a0e70 con 0x7f2110102290 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21101a0fe0 con 0x7f21101010a0 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21101a11a0 con 0x7f2110102c50 2024-03-11T18:32:43.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53256/0 (socket says 172.21.15.2:53256) 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210d400640 1 -- 172.21.15.2:0/2757975141 learned_addr learned my addr 172.21.15.2:0/2757975141 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56550/0 (socket says 172.21.15.2:56550) 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.888+0000 7f210e800640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2110102c50 0x7f21101a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 -- 172.21.15.2:0/2757975141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2110102c50 msgr2=0x7f21101a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2110102c50 0x7f21101a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 -- 172.21.15.2:0/2757975141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 msgr2=0x7f2110199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 -- 172.21.15.2:0/2757975141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f8009900 con 0x7f2110102290 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210de00640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f210d400640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20f8032710 tx=0x7f20f8034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8032e40 con 0x7f2110102290 2024-03-11T18:32:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.889+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20f8003e50 con 0x7f2110102290 2024-03-11T18:32:43.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.890+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21101a1420 con 0x7f2110102290 2024-03-11T18:32:43.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.890+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8040450 con 0x7f2110102290 2024-03-11T18:32:43.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.890+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21101a18a0 con 0x7f2110102290 2024-03-11T18:32:43.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.891+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211010cab0 con 0x7f2110102290 2024-03-11T18:32:43.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.893+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f20f800a040 con 0x7f2110102290 2024-03-11T18:32:43.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.894+0000 7f2106a00640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 0x7f20f00a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:43.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.894+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f20f80f02d0 con 0x7f2110102290 2024-03-11T18:32:43.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.895+0000 7f210de00640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 0x7f20f00a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:43.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.895+0000 7f210de00640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 0x7f20f00a3ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f21101020f0 tx=0x7f2100005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:43.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:43.899+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f20f80b91e0 con 0x7f2110102290 2024-03-11T18:32:44.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.151+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f211010cc40 con 0x7f2110102290 2024-03-11T18:32:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.153+0000 7f2106a00640 1 -- 172.21.15.2:0/2757975141 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f20f80bc8a0 con 0x7f2110102290 2024-03-11T18:32:44.155 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:44.156 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:32:44.156 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:32:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 msgr2=0x7f20f00a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 0x7f20f00a3ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f21101020f0 tx=0x7f2100005f90 comp rx=0 tx=0).stop 2024-03-11T18:32:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 msgr2=0x7f21101998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20f8032710 tx=0x7f20f8034cc0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 shutdown_connections 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f20f00a1a20 0x7f20f00a3ee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2110102c50 0x7f21101a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2110102290 0x7f21101998c0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20f8032710 tx=0x7f20f8034cc0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 --2- 172.21.15.2:0/2757975141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21101010a0 0x7f2110199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.155+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 >> 172.21.15.2:0/2757975141 conn(0x7f21100fc610 msgr2=0x7f21100733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.156+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 shutdown_connections 2024-03-11T18:32:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.156+0000 7f210f200640 1 -- 172.21.15.2:0/2757975141 wait complete. 2024-03-11T18:32:44.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:32:44.267 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:44 smithi002 ceph-mon[31328]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-03-11T18:32:44.267 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2857515821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.267 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:44 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/881301726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:44 smithi165 ceph-mon[36634]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-03-11T18:32:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:44 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2857515821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:44 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/881301726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:44 smithi023 ceph-mon[36406]: pgmap v392: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-03-11T18:32:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:44 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2857515821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:44 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/881301726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 -- 172.21.15.2:0/2045572714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104490 msgr2=0x7fd608104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 --2- 172.21.15.2:0/2045572714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104490 0x7fd608104910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009860 tx=0x7fd5f002bb70 comp rx=0 tx=0).stop 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 -- 172.21.15.2:0/2045572714 shutdown_connections 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 --2- 172.21.15.2:0/2045572714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd608104e50 0x7fd60810b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 --2- 172.21.15.2:0/2045572714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104490 0x7fd608104910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 --2- 172.21.15.2:0/2045572714 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6081032a0 0x7fd6081036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 -- 172.21.15.2:0/2045572714 >> 172.21.15.2:0/2045572714 conn(0x7fd6080fe830 msgr2=0x7fd608100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 -- 172.21.15.2:0/2045572714 shutdown_connections 2024-03-11T18:32:44.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.407+0000 7fd60e600640 1 -- 172.21.15.2:0/2045572714 wait complete. 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 Processor -- start 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 -- start start 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd608104490 0x7fd608199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 0x7fd6081a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6081a0e00 con 0x7fd608104490 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6081a0f70 con 0x7fd6081032a0 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.408+0000 7fd60e600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6081a1130 con 0x7fd608104e50 2024-03-11T18:32:44.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56580/0 (socket says 172.21.15.2:56580) 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 -- 172.21.15.2:0/3646338884 learned_addr learned my addr 172.21.15.2:0/3646338884 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60dc00640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 0x7fd6081a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd5ffe00640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd608104490 0x7fd608199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 -- 172.21.15.2:0/3646338884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 msgr2=0x7fd6081a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 0x7fd6081a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 -- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd608104490 msgr2=0x7fd608199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd608104490 0x7fd608199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60d200640 1 -- 172.21.15.2:0/3646338884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd600009900 con 0x7fd6081032a0 2024-03-11T18:32:44.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.409+0000 7fd60dc00640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 0x7fd6081a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd60d200640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800d770 tx=0x7fd5f800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8009c30 con 0x7fd6081032a0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f0009490 con 0x7fd6081032a0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5f800de80 con 0x7fd6081032a0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8010460 con 0x7fd6081032a0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.410+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6081a17b0 con 0x7fd6081032a0 2024-03-11T18:32:44.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.411+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60810a5c0 con 0x7fd6081032a0 2024-03-11T18:32:44.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.413+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd5f8003f60 con 0x7fd6081032a0 2024-03-11T18:32:44.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.415+0000 7fd606000640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 0x7fd5ec0a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.415+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd5f80cd460 con 0x7fd6081032a0 2024-03-11T18:32:44.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.415+0000 7fd5ffe00640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 0x7fd5ec0a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.416+0000 7fd5ffe00640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 0x7fd5ec0a3ee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd5f002c180 tx=0x7fd5f0034c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:44.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.417+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd5f8096360 con 0x7fd6081032a0 2024-03-11T18:32:44.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.620+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd60802cdd0 con 0x7fd6081032a0 2024-03-11T18:32:44.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.623+0000 7fd606000640 1 -- 172.21.15.2:0/3646338884 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7fd5f8017020 con 0x7fd6081032a0 2024-03-11T18:32:44.625 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:44.626 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:32:44.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 msgr2=0x7fd5ec0a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 0x7fd5ec0a3ee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd5f002c180 tx=0x7fd5f0034c00 comp rx=0 tx=0).stop 2024-03-11T18:32:44.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 msgr2=0x7fd608199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800d770 tx=0x7fd5f800dc30 comp rx=0 tx=0).stop 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 shutdown_connections 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd5ec0a1a20 0x7fd5ec0a3ee0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd608104e50 0x7fd6081a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd608104490 0x7fd608199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 --2- 172.21.15.2:0/3646338884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6081032a0 0x7fd608199310 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 >> 172.21.15.2:0/3646338884 conn(0x7fd6080fe830 msgr2=0x7fd608107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 shutdown_connections 2024-03-11T18:32:44.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.626+0000 7fd60e600640 1 -- 172.21.15.2:0/3646338884 wait complete. 2024-03-11T18:32:44.649 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:32:44.649 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:32:44.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.867+0000 7f89b1a00640 1 -- 172.21.15.2:0/3584711952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 msgr2=0x7f89ac103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.867+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3584711952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac103600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f899c009860 tx=0x7f899c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 -- 172.21.15.2:0/3584711952 shutdown_connections 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3584711952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac104d90 0x7f89ac10b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3584711952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac1043d0 0x7f89ac104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3584711952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac103600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 -- 172.21.15.2:0/3584711952 >> 172.21.15.2:0/3584711952 conn(0x7f89ac0fe750 msgr2=0x7f89ac100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 -- 172.21.15.2:0/3584711952 shutdown_connections 2024-03-11T18:32:44.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.868+0000 7f89b1a00640 1 -- 172.21.15.2:0/3584711952 wait complete. 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.869+0000 7f89b1a00640 1 Processor -- start 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.869+0000 7f89b1a00640 1 -- start start 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 0x7f89ac199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac104d90 0x7f89ac1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1a0df0 con 0x7f89ac1043d0 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1a0f60 con 0x7f89ac1031e0 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1a1120 con 0x7f89ac104d90 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89ab400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 0x7f89ac199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89ab400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 0x7f89ac199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53300/0 (socket says 172.21.15.2:53300) 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89ab400640 1 -- 172.21.15.2:0/3102953236 learned_addr learned my addr 172.21.15.2:0/3102953236 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89b1000640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac104d90 0x7f89ac1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 -- 172.21.15.2:0/3102953236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac104d90 msgr2=0x7f89ac1a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac104d90 0x7f89ac1a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 -- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 msgr2=0x7f89ac199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 0x7f89ac199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:44.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.870+0000 7f89abe00640 1 -- 172.21.15.2:0/3102953236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009900 con 0x7f89ac1031e0 2024-03-11T18:32:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89abe00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac199300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f899c009780 tx=0x7f899c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c02c510 con 0x7f89ac1031e0 2024-03-11T18:32:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f899c02cb50 con 0x7f89ac1031e0 2024-03-11T18:32:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c009490 con 0x7f89ac1031e0 2024-03-11T18:32:44.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c003850 con 0x7f89ac1031e0 2024-03-11T18:32:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.871+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac1a15d0 con 0x7f89ac1031e0 2024-03-11T18:32:44.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.872+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac10a5a0 con 0x7f89ac1031e0 2024-03-11T18:32:44.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.874+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f899c03b040 con 0x7f89ac1031e0 2024-03-11T18:32:44.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.876+0000 7f89a8c00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 0x7f89900a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:44.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.876+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f899c0ef290 con 0x7f89ac1031e0 2024-03-11T18:32:44.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.876+0000 7f89ab400640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 0x7f89900a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:44.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.877+0000 7f89ab400640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 0x7f89900a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f89ac19a810 tx=0x7f8994005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:44.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:44.880+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f899c0b8130 con 0x7f89ac1031e0 2024-03-11T18:32:45.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.083+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89ac02cdd0 con 0x7f89ac1031e0 2024-03-11T18:32:45.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.083+0000 7f89a8c00640 1 -- 172.21.15.2:0/3102953236 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f899c0bb7f0 con 0x7f89ac1031e0 2024-03-11T18:32:45.086 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 msgr2=0x7f89900a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 0x7f89900a3f70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f89ac19a810 tx=0x7f8994005f90 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 msgr2=0x7f89ac199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac199300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f899c009780 tx=0x7f899c005d50 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 shutdown_connections 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f89900a1ab0 0x7f89900a3f70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f89ac104d90 0x7f89ac1a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89ac1043d0 0x7f89ac199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 --2- 172.21.15.2:0/3102953236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89ac1031e0 0x7f89ac199300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 >> 172.21.15.2:0/3102953236 conn(0x7f89ac0fe750 msgr2=0x7f89ac107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:45.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 shutdown_connections 2024-03-11T18:32:45.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.086+0000 7f89b1a00640 1 -- 172.21.15.2:0/3102953236 wait complete. 2024-03-11T18:32:45.110 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:32:45.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.324+0000 7fec5e200640 1 -- 172.21.15.2:0/2678179429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102d40 msgr2=0x7fec581095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.324+0000 7fec5e200640 1 --2- 172.21.15.2:0/2678179429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102d40 0x7fec581095d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fec50009900 tx=0x7fec5002bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 -- 172.21.15.2:0/2678179429 shutdown_connections 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 --2- 172.21.15.2:0/2678179429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102d40 0x7fec581095d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 --2- 172.21.15.2:0/2678179429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102380 0x7fec58102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 --2- 172.21.15.2:0/2678179429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 0x7fec581015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 -- 172.21.15.2:0/2678179429 >> 172.21.15.2:0/2678179429 conn(0x7fec580fc700 msgr2=0x7fec580feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 -- 172.21.15.2:0/2678179429 shutdown_connections 2024-03-11T18:32:45.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.325+0000 7fec5e200640 1 -- 172.21.15.2:0/2678179429 wait complete. 2024-03-11T18:32:45.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 Processor -- start 2024-03-11T18:32:45.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 -- start start 2024-03-11T18:32:45.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 0x7fec58199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 0x7fec581a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581a0f40 con 0x7fec58102380 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581a10b0 con 0x7fec58102d40 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec5e200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581a1270 con 0x7fec58101190 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec56a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.326+0000 7fec57e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 0x7fec581a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec56a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53316/0 (socket says 172.21.15.2:53316) 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 0x7fec58199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec56a00640 1 -- 172.21.15.2:0/2220490734 learned_addr learned my addr 172.21.15.2:0/2220490734 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 -- 172.21.15.2:0/2220490734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 msgr2=0x7fec58199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 0x7fec58199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 -- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 msgr2=0x7fec58199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 -- 172.21.15.2:0/2220490734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50009560 con 0x7fec58102d40 2024-03-11T18:32:45.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec56a00640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.327+0000 7fec57e00640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 0x7fec581a0a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec50036040 tx=0x7fec50005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.328+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50004490 con 0x7fec58102d40 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.328+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec500052d0 con 0x7fec58102d40 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.328+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50003d90 con 0x7fec58102d40 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.328+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec581a14f0 con 0x7fec58102d40 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.328+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec581a19b0 con 0x7fec58102d40 2024-03-11T18:32:45.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.329+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5810cbf0 con 0x7fec58102d40 2024-03-11T18:32:45.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.331+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fec500035f0 con 0x7fec58102d40 2024-03-11T18:32:45.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.332+0000 7fec4fe00640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 0x7fec3c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.333+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fec500f0540 con 0x7fec58102d40 2024-03-11T18:32:45.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.333+0000 7fec57400640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 0x7fec3c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.333+0000 7fec57400640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 0x7fec3c0a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fec581021e0 tx=0x7fec48009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:45.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.334+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec500b9330 con 0x7fec58102d40 2024-03-11T18:32:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:45 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2757975141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:45 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3646338884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:45 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2757975141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:45 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3646338884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2757975141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:45 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3646338884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:45.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.536+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec581a1c90 con 0x7fec58102d40 2024-03-11T18:32:45.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.537+0000 7fec4fe00640 1 -- 172.21.15.2:0/2220490734 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec581a1c90 con 0x7fec58102d40 2024-03-11T18:32:45.539 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 msgr2=0x7fec3c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 0x7fec3c0a3ef0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fec581021e0 tx=0x7fec48009500 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 msgr2=0x7fec581a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 0x7fec581a0a00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec50036040 tx=0x7fec50005e20 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 shutdown_connections 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec3c0a1a30 0x7fec3c0a3ef0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec58102d40 0x7fec581a0a00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec58102380 0x7fec58199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 --2- 172.21.15.2:0/2220490734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58101190 0x7fec58199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 >> 172.21.15.2:0/2220490734 conn(0x7fec580fc700 msgr2=0x7fec580733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 shutdown_connections 2024-03-11T18:32:45.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.540+0000 7fec5e200640 1 -- 172.21.15.2:0/2220490734 wait complete. 2024-03-11T18:32:45.562 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:32:45.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 -- 172.21.15.2:0/2159545147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34106cb0 msgr2=0x7f3a34109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 --2- 172.21.15.2:0/2159545147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34106cb0 0x7f3a34109100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009900 tx=0x7f3a2c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:32:45.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 -- 172.21.15.2:0/2159545147 shutdown_connections 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 --2- 172.21.15.2:0/2159545147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34106cb0 0x7f3a34109100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 --2- 172.21.15.2:0/2159545147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34104380 0x7f3a34106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 --2- 172.21.15.2:0/2159545147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34101a50 0x7f3a34103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.791+0000 7f3a33200640 1 -- 172.21.15.2:0/2159545147 >> 172.21.15.2:0/2159545147 conn(0x7f3a340fb460 msgr2=0x7f3a340fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.792+0000 7f3a33200640 1 -- 172.21.15.2:0/2159545147 shutdown_connections 2024-03-11T18:32:45.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.792+0000 7f3a33200640 1 -- 172.21.15.2:0/2159545147 wait complete. 2024-03-11T18:32:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 Processor -- start 2024-03-11T18:32:45.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 -- start start 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 0x7f3a341013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 0x7f3a340ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34106cb0 0x7f3a340fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a341006a0 con 0x7f3a34101a50 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a34100810 con 0x7f3a34104380 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a33200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a341a1060 con 0x7f3a34106cb0 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a31e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 0x7f3a341013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a29400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 0x7f3a340ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a31e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 0x7f3a341013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53336/0 (socket says 172.21.15.2:53336) 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.793+0000 7f3a31e00640 1 -- 172.21.15.2:0/701882337 learned_addr learned my addr 172.21.15.2:0/701882337 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a32800640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34106cb0 0x7f3a340fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 -- 172.21.15.2:0/701882337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34106cb0 msgr2=0x7f3a340fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34106cb0 0x7f3a340fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 -- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 msgr2=0x7f3a341013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:45.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 0x7f3a341013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:45.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 -- 172.21.15.2:0/701882337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a24009840 con 0x7f3a34104380 2024-03-11T18:32:45.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a29400640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 0x7f3a340ffa00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00e9f0 tx=0x7f3a1c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.794+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00ccc0 con 0x7f3a34104380 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.795+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c009560 con 0x7f3a34104380 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.795+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c00ce20 con 0x7f3a34104380 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.795+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c004a00 con 0x7f3a34104380 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.795+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a341a1690 con 0x7f3a34104380 2024-03-11T18:32:45.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.796+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a340fcf50 con 0x7f3a34104380 2024-03-11T18:32:45.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.798+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f3a1c0050b0 con 0x7f3a34104380 2024-03-11T18:32:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.799+0000 7f3a2b400640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 0x7f3a180a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.799+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f3a1c014070 con 0x7f3a34104380 2024-03-11T18:32:45.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.800+0000 7f3a31e00640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 0x7f3a180a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.801+0000 7f3a31e00640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 0x7f3a180a3f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3a24011630 tx=0x7f3a24009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:45.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:45.801+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3a1c096600 con 0x7f3a34104380 2024-03-11T18:32:46.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.001+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a3410cc80 con 0x7f3a34104380 2024-03-11T18:32:46.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.001+0000 7f3a2b400640 1 -- 172.21.15.2:0/701882337 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a1c099cc0 con 0x7f3a34104380 2024-03-11T18:32:46.004 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 msgr2=0x7f3a180a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 0x7f3a180a3f70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3a24011630 tx=0x7f3a24009500 comp rx=0 tx=0).stop 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 msgr2=0x7f3a340ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 0x7f3a340ffa00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00e9f0 tx=0x7f3a1c00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 shutdown_connections 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f3a180a1ab0 0x7f3a180a3f70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a34106cb0 0x7f3a340fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a34104380 0x7f3a340ffa00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 --2- 172.21.15.2:0/701882337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a34101a50 0x7f3a341013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 >> 172.21.15.2:0/701882337 conn(0x7f3a340fb460 msgr2=0x7f3a34102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 shutdown_connections 2024-03-11T18:32:46.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.004+0000 7f3a33200640 1 -- 172.21.15.2:0/701882337 wait complete. 2024-03-11T18:32:46.026 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 -- 172.21.15.2:0/926830505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 msgr2=0x7f1144106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 --2- 172.21.15.2:0/926830505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144106770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1130009860 tx=0x7f113002bb70 comp rx=0 tx=0).stop 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 -- 172.21.15.2:0/926830505 shutdown_connections 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 --2- 172.21.15.2:0/926830505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f11441090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 --2- 172.21.15.2:0/926830505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144106770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 --2- 172.21.15.2:0/926830505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 0x7f11440fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 -- 172.21.15.2:0/926830505 >> 172.21.15.2:0/926830505 conn(0x7f11440fa780 msgr2=0x7f11440fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 -- 172.21.15.2:0/926830505 shutdown_connections 2024-03-11T18:32:46.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.240+0000 7f114a400640 1 -- 172.21.15.2:0/926830505 wait complete. 2024-03-11T18:32:46.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.241+0000 7f114a400640 1 Processor -- start 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.241+0000 7f114a400640 1 -- start start 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 0x7f1144102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11441013d0 con 0x7f1144106cb0 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144101540 con 0x7f11440feba0 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f114a400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11441a1130 con 0x7f11440ff500 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53354/0 (socket says 172.21.15.2:53354) 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149a00640 1 -- 172.21.15.2:0/35713775 learned_addr learned my addr 172.21.15.2:0/35713775 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149000640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 0x7f1144102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1143e00640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149a00640 1 -- 172.21.15.2:0/35713775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 msgr2=0x7f1144100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.242+0000 7f1149a00640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1149a00640 1 -- 172.21.15.2:0/35713775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 msgr2=0x7f1144102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1149a00640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 0x7f1144102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1149a00640 1 -- 172.21.15.2:0/35713775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1138009560 con 0x7f1144106cb0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1149a00640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f113c00eb40 tx=0x7f113c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c004b20 con 0x7f1144106cb0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f113c004c80 con 0x7f1144106cb0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1130009490 con 0x7f1144106cb0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c010670 con 0x7f1144106cb0 2024-03-11T18:32:46.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.243+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11441a16e0 con 0x7f1144106cb0 2024-03-11T18:32:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.244+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144105580 con 0x7f1144106cb0 2024-03-11T18:32:46.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.246+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f113c005020 con 0x7f1144106cb0 2024-03-11T18:32:46.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.248+0000 7f1141600640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 0x7f112c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.248+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f113c014070 con 0x7f1144106cb0 2024-03-11T18:32:46.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.248+0000 7f1149000640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 0x7f112c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.249+0000 7f1149000640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 0x7f112c0a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1138009530 tx=0x7f1138009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.252+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f113c096b20 con 0x7f1144106cb0 2024-03-11T18:32:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:46 smithi023 ceph-mon[36406]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3102953236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2220490734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:46 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/701882337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:46 smithi165 ceph-mon[36634]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3102953236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2220490734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:46 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/701882337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.452+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f114402ce40 con 0x7f1144106cb0 2024-03-11T18:32:46.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.453+0000 7f1141600640 1 -- 172.21.15.2:0/35713775 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7f113c09a1e0 con 0x7f1144106cb0 2024-03-11T18:32:46.455 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:32:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.455+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 msgr2=0x7f112c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.455+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 0x7f112c0a3f70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1138009530 tx=0x7f1138009260 comp rx=0 tx=0).stop 2024-03-11T18:32:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.455+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 msgr2=0x7f1144100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.455+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f113c00eb40 tx=0x7f113c00c2d0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 shutdown_connections 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f112c0a1ab0 0x7f112c0a3f70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1144106cb0 0x7f1144100c40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f11440ff500 0x7f1144100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 --2- 172.21.15.2:0/35713775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11440feba0 0x7f1144102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 >> 172.21.15.2:0/35713775 conn(0x7f11440fa780 msgr2=0x7f1144102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 shutdown_connections 2024-03-11T18:32:46.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.456+0000 7f114a400640 1 -- 172.21.15.2:0/35713775 wait complete. 2024-03-11T18:32:46.474 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:46 smithi002 ceph-mon[31328]: pgmap v393: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:46.474 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3102953236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.475 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2220490734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.475 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:46 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/701882337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:32:46.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 -- 172.21.15.2:0/2540981430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 msgr2=0x7efe84077fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 --2- 172.21.15.2:0/2540981430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe84077fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009be0 tx=0x7efe7c02bef0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 -- 172.21.15.2:0/2540981430 shutdown_connections 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 --2- 172.21.15.2:0/2540981430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe84077fc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 --2- 172.21.15.2:0/2540981430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84076d80 0x7efe840771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 --2- 172.21.15.2:0/2540981430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84075b40 0x7efe84075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 -- 172.21.15.2:0/2540981430 >> 172.21.15.2:0/2540981430 conn(0x7efe840fe830 msgr2=0x7efe84100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 -- 172.21.15.2:0/2540981430 shutdown_connections 2024-03-11T18:32:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.689+0000 7efe83200640 1 -- 172.21.15.2:0/2540981430 wait complete. 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 Processor -- start 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 -- start start 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 0x7efe8419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 0x7efe8419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe841a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe841a5150 con 0x7efe84075b40 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe841a52c0 con 0x7efe84077720 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.690+0000 7efe83200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe841a5480 con 0x7efe84076d80 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 0x7efe8419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 0x7efe8419d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53376/0 (socket says 172.21.15.2:53376) 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81e00640 1 -- 172.21.15.2:0/1967704127 learned_addr learned my addr 172.21.15.2:0/1967704127 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 0x7efe8419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe82800640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe841a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 -- 172.21.15.2:0/1967704127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 msgr2=0x7efe841a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe841a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 -- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 msgr2=0x7efe8419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 0x7efe8419d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.691+0000 7efe81400640 1 -- 172.21.15.2:0/1967704127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70009560 con 0x7efe84076d80 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe81400640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 0x7efe8419dba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe70005fd0 tx=0x7efe7000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000cd00 con 0x7efe84076d80 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c009840 con 0x7efe84076d80 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe7000ce60 con 0x7efe84076d80 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70010430 con 0x7efe84076d80 2024-03-11T18:32:46.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.692+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe841a5b30 con 0x7efe84076d80 2024-03-11T18:32:46.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.693+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8410fc10 con 0x7efe84076d80 2024-03-11T18:32:46.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.696+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7efe70003760 con 0x7efe84076d80 2024-03-11T18:32:46.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.697+0000 7efe7aa00640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 0x7efe680a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:46.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.698+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7efe70014070 con 0x7efe84076d80 2024-03-11T18:32:46.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.698+0000 7efe81e00640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 0x7efe680a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:46.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.698+0000 7efe81e00640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 0x7efe680a3ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efe74009c90 tx=0x7efe74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:46.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.699+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efe70096950 con 0x7efe84076d80 2024-03-11T18:32:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.941+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe841130f0 con 0x7efe84076d80 2024-03-11T18:32:46.946 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:46.946 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:32:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.942+0000 7efe7aa00640 1 -- 172.21.15.2:0/1967704127 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7efe7009a010 con 0x7efe84076d80 2024-03-11T18:32:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 msgr2=0x7efe680a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 0x7efe680a3ee0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efe74009c90 tx=0x7efe74005f90 comp rx=0 tx=0).stop 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 msgr2=0x7efe8419dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 0x7efe8419dba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe70005fd0 tx=0x7efe7000ee50 comp rx=0 tx=0).stop 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 shutdown_connections 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7efe680a1a20 0x7efe680a3ee0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe84077720 0x7efe841a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe84076d80 0x7efe8419dba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 --2- 172.21.15.2:0/1967704127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efe84075b40 0x7efe8419d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 >> 172.21.15.2:0/1967704127 conn(0x7efe840fe830 msgr2=0x7efe84100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 shutdown_connections 2024-03-11T18:32:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:46.946+0000 7efe83200640 1 -- 172.21.15.2:0/1967704127 wait complete. 2024-03-11T18:32:46.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 -- 172.21.15.2:0/1166101659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104490 msgr2=0x7f6db0104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 --2- 172.21.15.2:0/1166101659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104490 0x7f6db0104910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009900 tx=0x7f6d9c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 -- 172.21.15.2:0/1166101659 shutdown_connections 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 --2- 172.21.15.2:0/1166101659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db0104e50 0x7f6db010b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 --2- 172.21.15.2:0/1166101659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104490 0x7f6db0104910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 --2- 172.21.15.2:0/1166101659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db01032a0 0x7f6db01036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 -- 172.21.15.2:0/1166101659 >> 172.21.15.2:0/1166101659 conn(0x7f6db00fe830 msgr2=0x7f6db0100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 -- 172.21.15.2:0/1166101659 shutdown_connections 2024-03-11T18:32:47.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.194+0000 7f6db7200640 1 -- 172.21.15.2:0/1166101659 wait complete. 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.195+0000 7f6db7200640 1 Processor -- start 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 -- start start 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db0104490 0x7f6db0199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104e50 0x7f6db01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db01a0dd0 con 0x7f6db01032a0 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db01a0f40 con 0x7f6db0104490 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db01a1100 con 0x7f6db0104e50 2024-03-11T18:32:47.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40660/0 (socket says 172.21.15.2:40660) 2024-03-11T18:32:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db5e00640 1 -- 172.21.15.2:0/2158763202 learned_addr learned my addr 172.21.15.2:0/2158763202 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.196+0000 7f6db5400640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db0104490 0x7f6db0199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db6800640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104e50 0x7f6db01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 -- 172.21.15.2:0/2158763202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104e50 msgr2=0x7f6db01a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104e50 0x7f6db01a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 -- 172.21.15.2:0/2158763202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db0104490 msgr2=0x7f6db0199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db0104490 0x7f6db0199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 -- 172.21.15.2:0/2158763202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c009560 con 0x7f6db01032a0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.197+0000 7f6db5e00640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6da400e9c0 tx=0x7f6da400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.198+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4009700 con 0x7f6db01032a0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.198+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da400cd80 con 0x7f6db01032a0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.198+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4010430 con 0x7f6db01032a0 2024-03-11T18:32:47.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.198+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db01a13e0 con 0x7f6db01032a0 2024-03-11T18:32:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.198+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db01a18c0 con 0x7f6db01032a0 2024-03-11T18:32:47.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.199+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db010a560 con 0x7f6db01032a0 2024-03-11T18:32:47.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.201+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f6da4003680 con 0x7f6db01032a0 2024-03-11T18:32:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.203+0000 7f6daaa00640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 0x7f6d980a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.203+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f6da4014070 con 0x7f6db01032a0 2024-03-11T18:32:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.203+0000 7f6db5400640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 0x7f6d980a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.204+0000 7f6db5400640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 0x7f6d980a3ee0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c000c00 tx=0x7f6d9c02c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:47.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.208+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6da40969c0 con 0x7f6db01032a0 2024-03-11T18:32:47.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:47 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/35713775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:47.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:47 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1967704127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:47.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:47 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/35713775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:47.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:47 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1967704127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:47.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:47 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/35713775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:32:47.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:47 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1967704127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:47.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.411+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6db010ed00 con 0x7f6db01032a0 2024-03-11T18:32:47.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.414+0000 7f6daaa00640 1 -- 172.21.15.2:0/2158763202 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7f6da409a080 con 0x7f6db01032a0 2024-03-11T18:32:47.417 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:47.418 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 msgr2=0x7f6d980a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 0x7f6d980a3ee0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c000c00 tx=0x7f6d9c02c7f0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 msgr2=0x7f6db01992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6da400e9c0 tx=0x7f6da400ee80 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 shutdown_connections 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f6d980a1a20 0x7f6d980a3ee0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6db0104e50 0x7f6db01a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6db0104490 0x7f6db0199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 --2- 172.21.15.2:0/2158763202 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6db01032a0 0x7f6db01992e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 >> 172.21.15.2:0/2158763202 conn(0x7f6db00fe830 msgr2=0x7f6db0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:47.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 shutdown_connections 2024-03-11T18:32:47.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.417+0000 7f6db7200640 1 -- 172.21.15.2:0/2158763202 wait complete. 2024-03-11T18:32:47.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 -- 172.21.15.2:0/3306102523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8104630 msgr2=0x7f78e8104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 --2- 172.21.15.2:0/3306102523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8104630 0x7f78e8104ab0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009900 tx=0x7f78d802bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 -- 172.21.15.2:0/3306102523 shutdown_connections 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 --2- 172.21.15.2:0/3306102523 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104ff0 0x7f78e810b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 --2- 172.21.15.2:0/3306102523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8104630 0x7f78e8104ab0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 --2- 172.21.15.2:0/3306102523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8103440 0x7f78e8103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 -- 172.21.15.2:0/3306102523 >> 172.21.15.2:0/3306102523 conn(0x7f78e80fe990 msgr2=0x7f78e8100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 -- 172.21.15.2:0/3306102523 shutdown_connections 2024-03-11T18:32:47.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.657+0000 7f78f0800640 1 -- 172.21.15.2:0/3306102523 wait complete. 2024-03-11T18:32:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 Processor -- start 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 -- start start 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 0x7f78e8199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8104ff0 0x7f78e81a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e81a0f00 con 0x7f78e8104ff0 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e81a1070 con 0x7f78e8104630 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.658+0000 7f78f0800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e81a1230 con 0x7f78e8103440 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ed400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ed400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45806/0 (socket says 172.21.15.2:45806) 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ed400640 1 -- 172.21.15.2:0/2172270110 learned_addr learned my addr 172.21.15.2:0/2172270110 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 0x7f78e8199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ee800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8104ff0 0x7f78e81a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 -- 172.21.15.2:0/2172270110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 msgr2=0x7f78e8199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 -- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8104ff0 msgr2=0x7f78e81a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8104ff0 0x7f78e81a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 -- 172.21.15.2:0/2172270110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8009560 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ed400640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.659+0000 7f78ede00640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 0x7f78e8199410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9c0 tx=0x7f78dc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.660+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc00ccc0 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.660+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78dc00ce20 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.660+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc010430 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.660+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e81a1510 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.660+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e81a1a70 con 0x7f78e8103440 2024-03-11T18:32:47.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.661+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e810a700 con 0x7f78e8103440 2024-03-11T18:32:47.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.663+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f78dc003f60 con 0x7f78e8103440 2024-03-11T18:32:47.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.664+0000 7f78e2a00640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 0x7f78d00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:47.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.665+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f78dc014070 con 0x7f78e8103440 2024-03-11T18:32:47.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.665+0000 7f78ed400640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 0x7f78d00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:47.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.665+0000 7f78ed400640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 0x7f78d00a3f70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f78e819a920 tx=0x7f78d802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:47.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.667+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f78dc096a20 con 0x7f78e8103440 2024-03-11T18:32:47.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.904+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78e810ede0 con 0x7f78e8103440 2024-03-11T18:32:47.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.905+0000 7f78e2a00640 1 -- 172.21.15.2:0/2172270110 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f78dc09a0e0 con 0x7f78e8103440 2024-03-11T18:32:47.909 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:32:47.909 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:32:47.909 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 msgr2=0x7f78d00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 0x7f78d00a3f70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f78e819a920 tx=0x7f78d802c7e0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 msgr2=0x7f78e8199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 0x7f78e8199410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9c0 tx=0x7f78dc00ee80 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 shutdown_connections 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f78d00a1ab0 0x7f78d00a3f70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f78e8104ff0 0x7f78e81a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78e8104630 0x7f78e8199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 --2- 172.21.15.2:0/2172270110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f78e8103440 0x7f78e8199410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 >> 172.21.15.2:0/2172270110 conn(0x7f78e80fe990 msgr2=0x7f78e81078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 shutdown_connections 2024-03-11T18:32:47.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:47.908+0000 7f78f0800640 1 -- 172.21.15.2:0/2172270110 wait complete. 2024-03-11T18:32:47.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T18:30:27.790111+0000', 'modified': '2024-03-11T18:31:01.455980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24550, 'mds_2': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6835/1605486605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6834', 'nonce': 1605486605}, {'type': 'v1', 'addr': '172.21.15.23:6835', 'nonce': 1605486605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6835/2758205355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6834', 'nonce': 2758205355}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 2758205355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6837/3141279332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6836', 'nonce': 3141279332}, {'type': 'v1', 'addr': '172.21.15.165:6837', 'nonce': 3141279332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.23:6837/531049865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6836', 'nonce': 531049865}, {'type': 'v1', 'addr': '172.21.15.23:6837', 'nonce': 531049865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.165:6839/1329073865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6838', 'nonce': 1329073865}, {'type': 'v1', 'addr': '172.21.15.165:6839', 'nonce': 1329073865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.23:6839/2250151518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 2250151518}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 2250151518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T18:32:47.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-11T18:32:47.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T18:32:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.146+0000 7f18ea200640 1 -- 172.21.15.2:0/3339930893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e4102d80 msgr2=0x7f18e4109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.146+0000 7f18ea200640 1 --2- 172.21.15.2:0/3339930893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e4102d80 0x7f18e4109610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009900 tx=0x7f18dc02bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 -- 172.21.15.2:0/3339930893 shutdown_connections 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 --2- 172.21.15.2:0/3339930893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e4102d80 0x7f18e4109610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 --2- 172.21.15.2:0/3339930893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e41023c0 0x7f18e4102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 --2- 172.21.15.2:0/3339930893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41011d0 0x7f18e41015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 -- 172.21.15.2:0/3339930893 >> 172.21.15.2:0/3339930893 conn(0x7f18e40fc740 msgr2=0x7f18e40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 -- 172.21.15.2:0/3339930893 shutdown_connections 2024-03-11T18:32:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.147+0000 7f18ea200640 1 -- 172.21.15.2:0/3339930893 wait complete. 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.148+0000 7f18ea200640 1 Processor -- start 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.148+0000 7f18ea200640 1 -- start start 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e41011d0 0x7f18e4199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4102d80 0x7f18e41a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a0f10 con 0x7f18e4102d80 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a1080 con 0x7f18e41023c0 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18ea200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e41a1240 con 0x7f18e41011d0 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4102d80 0x7f18e41a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45824/0 (socket says 172.21.15.2:45824) 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 -- 172.21.15.2:0/3155872123 learned_addr learned my addr 172.21.15.2:0/3155872123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e3400640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e41011d0 0x7f18e4199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 -- 172.21.15.2:0/3155872123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e41011d0 msgr2=0x7f18e4199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e41011d0 0x7f18e4199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 -- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4102d80 msgr2=0x7f18e41a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4102d80 0x7f18e41a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.149+0000 7f18e2a00640 1 -- 172.21.15.2:0/3155872123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc009560 con 0x7f18e41023c0 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18e2a00640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00d950 tx=0x7f18cc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18dbe00640 1 -- 172.21.15.2:0/3155872123 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc009a30 con 0x7f18e41023c0 2024-03-11T18:32:48.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e41a1520 con 0x7f18e41023c0 2024-03-11T18:32:48.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18dbe00640 1 -- 172.21.15.2:0/3155872123 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18cc009b90 con 0x7f18e41023c0 2024-03-11T18:32:48.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18dbe00640 1 -- 172.21.15.2:0/3155872123 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc004a00 con 0x7f18e41023c0 2024-03-11T18:32:48.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.150+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e41a1a30 con 0x7f18e41023c0 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.153+0000 7f18ecf7a740 1 Processor -- start 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.153+0000 7f18dbe00640 1 -- 172.21.15.2:0/3155872123 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f18cc009d00 con 0x7f18e41023c0 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 -- start start 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 0x559c94e66a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e670e0 0x559c94e69570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94dd0b30 con 0x559c94e61580 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94dce060 con 0x559c94e61f70 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94dc9840 con 0x559c94e670e0 2024-03-11T18:32:48.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40712/0 (socket says 172.21.15.2:40712) 2024-03-11T18:32:48.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18e3e00640 1 -- 172.21.15.2:0/2840134945 learned_addr learned my addr 172.21.15.2:0/2840134945 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18e2a00640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e670e0 0x559c94e69570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.154+0000 7f18e3400640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 0x559c94e66a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 -- 172.21.15.2:0/2840134945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e670e0 msgr2=0x559c94e69570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e670e0 0x559c94e69570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 -- 172.21.15.2:0/2840134945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 msgr2=0x559c94e66a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 0x559c94e66a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 -- 172.21.15.2:0/2840134945 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c94d914c0 con 0x559c94e61580 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3400640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 0x559c94e66a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18e3e00640 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009900 tx=0x7f18dc005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5499aba61742ed5c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.155+0000 7f18dbe00640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 0x7f18c80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.156+0000 7f18d3e00640 1 -- 172.21.15.2:0/2840134945 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc02cb20 con 0x559c94e61580 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.156+0000 7f18dbe00640 1 -- 172.21.15.2:0/3155872123 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f18cc0cd1a0 con 0x7f18e41023c0 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.156+0000 7f18d3e00640 1 -- 172.21.15.2:0/2840134945 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18dc02cc80 con 0x559c94e61580 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.156+0000 7f18d3e00640 1 -- 172.21.15.2:0/2840134945 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc0052f0 con 0x559c94e61580 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.156+0000 7f18e3400640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 0x7f18c80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18e3400640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 0x7f18c80a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x559c94e62370 tx=0x7f18d4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- 172.21.15.2:0/2840134945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 msgr2=0x559c94e619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009900 tx=0x7f18dc005e60 comp rx=0 tx=0).stop 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- 172.21.15.2:0/2840134945 shutdown_connections 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e670e0 0x559c94e69570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61f70 0x559c94e66a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/2840134945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61580 0x559c94e619a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- 172.21.15.2:0/2840134945 >> 172.21.15.2:0/2840134945 conn(0x559c94e5caf0 msgr2=0x559c94e5cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- 172.21.15.2:0/2840134945 shutdown_connections 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- 172.21.15.2:0/2840134945 wait complete. 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.157+0000 7f18ecf7a740 1 -- start start 2024-03-11T18:32:48.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 Processor -- start 2024-03-11T18:32:48.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e6b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e72920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94d6eea0 con 0x559c94e670e0 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94d6eb40 con 0x559c94e61580 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.158+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94b11f20 con 0x559c94e61f70 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18ecf7a740 10 client.0 _ll_register_callbacks cb 0x559c94d82950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45836/0 (socket says 172.21.15.2:45836) 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e72920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 -- 172.21.15.2:0/1895024255 learned_addr learned my addr 172.21.15.2:0/1895024255 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e6b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 -- 172.21.15.2:0/1895024255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 msgr2=0x559c94e6b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e6b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 -- 172.21.15.2:0/1895024255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 msgr2=0x559c94e72920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e72920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3400640 1 -- 172.21.15.2:0/1895024255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e4109f10 con 0x559c94e61580 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e6b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.159+0000 7f18e3e00640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e72920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18e3400640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18d400fab0 tx=0x7f18d4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc9345c6df49cf25 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d4016070 con 0x559c94e61580 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f18d4004b20 con 0x559c94e61580 2024-03-11T18:32:48.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c94ac96c0 con 0x559c94e61580 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18d4004ce0 con 0x559c94e61580 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18ecf7a740 10 client.25157 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.160+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559c94dc7ec0 con 0x559c94e61580 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559c94ac96c0 con 0x559c94e61580 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f18d401ea00 con 0x559c94e61580 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 10 client.25157 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 10 client.25157 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 20 client.25157 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94e76c60 0x559c94e7bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 4 client.25157 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 --> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c94b1a270 con 0x559c94e76c60 2024-03-11T18:32:48.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.161+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94e76c60 0x559c94e7bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.162+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94e76c60 0x559c94e7bcc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7d211d8c1de4f89 server_cookie=83a0f8c06bd47a0f in_seq=0 out_seq=0 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.162+0000 7f18d0c00640 10 client.25157 ms_handle_connect on v2:172.21.15.23:6836/531049865 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18d0c00640 1 -- 172.21.15.2:0/1895024255 <== mds.2 v2:172.21.15.23:6836/531049865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559c94b1a270 con 0x559c94e76c60 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18d0c00640 10 client.25157 handle_command_reply: tid=0 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 client.25157 shutdown 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 msgr2=0x559c94e6b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18d400fab0 tx=0x7f18d4004ab0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 shutdown_connections 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94e76c60 0x559c94e7bcc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e72920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e6b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18d0c00640 0 client.25157 ms_handle_reset on v2:172.21.15.23:6836/531049865 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1895024255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e6b3a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 >> 172.21.15.2:0/1895024255 conn(0x559c94e5caf0 msgr2=0x559c94e5be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 shutdown_connections 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.163+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1895024255 wait complete. 2024-03-11T18:32:48.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.164+0000 7f18ecf7a740 20 client.25157 trim_cache size 0 max 16384 2024-03-11T18:32:48.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 Processor -- start 2024-03-11T18:32:48.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 -- start start 2024-03-11T18:32:48.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e5b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94e731e0 con 0x559c94e670e0 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94e79ad0 con 0x559c94e61580 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.185+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94e79820 con 0x559c94e61f70 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40742/0 (socket says 172.21.15.2:40742) 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e5b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 -- 172.21.15.2:0/452312851 learned_addr learned my addr 172.21.15.2:0/452312851 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e2a00640 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 -- 172.21.15.2:0/452312851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 msgr2=0x559c94e728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 -- 172.21.15.2:0/452312851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 msgr2=0x559c94e5b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e5b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3e00640 1 -- 172.21.15.2:0/452312851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c94d648b0 con 0x559c94e670e0 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.186+0000 7f18e3400640 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e5b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.187+0000 7f18e3e00640 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009900 tx=0x7f18dc004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=620e5c55b1681e0a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.187+0000 7f18d3400640 1 -- 172.21.15.2:0/452312851 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc02cb20 con 0x559c94e670e0 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.187+0000 7f18d3400640 1 -- 172.21.15.2:0/452312851 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18dc02cc80 con 0x559c94e670e0 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.187+0000 7f18d3400640 1 -- 172.21.15.2:0/452312851 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc003850 con 0x559c94e670e0 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- 172.21.15.2:0/452312851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 msgr2=0x559c94e5bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009900 tx=0x7f18dc004df0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- 172.21.15.2:0/452312851 shutdown_connections 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e670e0 0x559c94e5bac0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61f70 0x559c94e728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/452312851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e61580 0x559c94e5b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- 172.21.15.2:0/452312851 >> 172.21.15.2:0/452312851 conn(0x559c94e5caf0 msgr2=0x559c94e80570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- 172.21.15.2:0/452312851 shutdown_connections 2024-03-11T18:32:48.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- 172.21.15.2:0/452312851 wait complete. 2024-03-11T18:32:48.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.188+0000 7f18ecf7a740 1 -- start start 2024-03-11T18:32:48.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 Processor -- start 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 0x559c94e72600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 0x559c94e6f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94e6fc60 con 0x559c94e61f70 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94e6fdd0 con 0x559c94e670e0 2024-03-11T18:32:48.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c94ecd980 con 0x559c94e61580 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18ecf7a740 10 client.0 _ll_register_callbacks cb 0x7f18d40114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18e2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18e2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40750/0 (socket says 172.21.15.2:40750) 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18e2a00640 1 -- 172.21.15.2:0/1626082647 learned_addr learned my addr 172.21.15.2:0/1626082647 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.189+0000 7f18e3400640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 0x559c94e72600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e3e00640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 0x559c94e6f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 -- 172.21.15.2:0/1626082647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 msgr2=0x559c94e72600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 0x559c94e72600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 -- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 msgr2=0x559c94e6f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 0x559c94e6f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 -- 172.21.15.2:0/1626082647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c94dc7ec0 con 0x559c94e61f70 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e3e00640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 0x559c94e6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e3400640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 0x559c94e72600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18e2a00640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0966c0 tx=0x7f18cc004720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433f9e0d7e45e3fb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.190+0000 7f18d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc099cf0 con 0x559c94e61f70 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f18cc09de40 con 0x559c94e61f70 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c94dedc40 con 0x559c94e61f70 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18cc0a2490 con 0x559c94e61f70 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18ecf7a740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559c94d914c0 con 0x559c94e61f70 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559c94dedc40 con 0x559c94e61f70 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f18cc0b80e0 con 0x559c94e61f70 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.191+0000 7f18ecf7a740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18ecf7a740 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18ecf7a740 20 client.15066 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94ed2c30 0x559c94ecf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18ecf7a740 4 client.15066 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 --> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559c94b06610 con 0x559c94ed2c30 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18e3400640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94ed2c30 0x559c94ecf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18e3400640 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94ed2c30 0x559c94ecf070 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0e4851eb2306caa server_cookie=81cc54299423e12e in_seq=0 out_seq=0 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.192+0000 7f18d0c00640 10 client.15066 ms_handle_connect on v2:172.21.15.23:6836/531049865 2024-03-11T18:32:48.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18d0c00640 1 -- 172.21.15.2:0/1626082647 <== mds.2 v2:172.21.15.23:6836/531049865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559c94b06610 con 0x559c94ed2c30 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18d0c00640 10 client.15066 handle_command_reply: tid=0 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 client.15066 shutdown 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 msgr2=0x559c94e6f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f18cc0966c0 tx=0x7f18cc004720 comp rx=0 tx=0).stop 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 shutdown_connections 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x559c94ed2c30 0x559c94ecf070 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c94e670e0 0x559c94e6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559c94e61f70 0x559c94e6f1e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 --2- 172.21.15.2:0/1626082647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559c94e61580 0x559c94e72600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18d0c00640 0 client.15066 ms_handle_reset on v2:172.21.15.23:6836/531049865 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 >> 172.21.15.2:0/1626082647 conn(0x559c94e5caf0 msgr2=0x559c94e5fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 shutdown_connections 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 1 -- 172.21.15.2:0/1626082647 wait complete. 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.193+0000 7f18ecf7a740 20 client.15066 trim_cache size 0 max 16384 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24521, 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:32:48.202 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 107.745400315, 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 160.29980501399999 2024-03-11T18:32:48.203 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 msgr2=0x7f18c80a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 0x7f18c80a3ef0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x559c94e62370 tx=0x7f18d4009500 comp rx=0 tx=0).stop 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 msgr2=0x7f18e4199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00d950 tx=0x7f18cc00de10 comp rx=0 tx=0).stop 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 shutdown_connections 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f18c80a1a30 0x7f18c80a3ef0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18e4102d80 0x7f18e41a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18e41023c0 0x7f18e4199960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 --2- 172.21.15.2:0/3155872123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18e41011d0 0x7f18e4199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 >> 172.21.15.2:0/3155872123 conn(0x7f18e40fc740 msgr2=0x7f18e40733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 shutdown_connections 2024-03-11T18:32:48.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.194+0000 7f18ea200640 1 -- 172.21.15.2:0/3155872123 wait complete. 2024-03-11T18:32:48.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:48 smithi002 ceph-mon[31328]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:48.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:48 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2158763202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:48 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2172270110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.217 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T18:32:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:48 smithi165 ceph-mon[36634]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:48 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2158763202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:48 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2172270110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:48 smithi023 ceph-mon[36406]: pgmap v394: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2024-03-11T18:32:48.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:48 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2158763202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:48 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2172270110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:32:48.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.436+0000 7f596e400640 1 -- 172.21.15.2:0/1064812054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5968109370 msgr2=0x7f596810fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.436+0000 7f596e400640 1 --2- 172.21.15.2:0/1064812054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5968109370 0x7f596810fc00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5960009be0 tx=0x7f596002bef0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 -- 172.21.15.2:0/1064812054 shutdown_connections 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 --2- 172.21.15.2:0/1064812054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5968109370 0x7f596810fc00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 --2- 172.21.15.2:0/1064812054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f5968108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 --2- 172.21.15.2:0/1064812054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59681077c0 0x7f5968107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 -- 172.21.15.2:0/1064812054 >> 172.21.15.2:0/1064812054 conn(0x7f5968076770 msgr2=0x7f5968078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 -- 172.21.15.2:0/1064812054 shutdown_connections 2024-03-11T18:32:48.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.437+0000 7f596e400640 1 -- 172.21.15.2:0/1064812054 wait complete. 2024-03-11T18:32:48.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 Processor -- start 2024-03-11T18:32:48.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 -- start start 2024-03-11T18:32:48.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 0x7f596819d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f596819dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681a5320 con 0x7f5968109370 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681a5490 con 0x7f59681077c0 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.438+0000 7f596e400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681a5650 con 0x7f59681089b0 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 0x7f596819d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5966a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f596819dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40770/0 (socket says 172.21.15.2:40770) 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 -- 172.21.15.2:0/3974107179 learned_addr learned my addr 172.21.15.2:0/3974107179 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 0x7f596819d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45868/0 (socket says 172.21.15.2:45868) 2024-03-11T18:32:48.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 -- 172.21.15.2:0/3974107179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 msgr2=0x7f596819dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f596819dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 -- 172.21.15.2:0/3974107179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 msgr2=0x7f596819d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 0x7f596819d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 -- 172.21.15.2:0/3974107179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5958009560 con 0x7f5968109370 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5966a00640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f596819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.439+0000 7f5967e00640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5960033420 tx=0x7f5960033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.440+0000 7f595fe00640 1 -- 172.21.15.2:0/3974107179 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960003b80 con 0x7f5968109370 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.440+0000 7f595fe00640 1 -- 172.21.15.2:0/3974107179 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5960003ce0 con 0x7f5968109370 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.440+0000 7f595fe00640 1 -- 172.21.15.2:0/3974107179 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59600348c0 con 0x7f5968109370 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.440+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5960009840 con 0x7f5968109370 2024-03-11T18:32:48.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.440+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59681a5bf0 con 0x7f5968109370 2024-03-11T18:32:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.443+0000 7f59711cc740 1 Processor -- start 2024-03-11T18:32:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.443+0000 7f595fe00640 1 -- 172.21.15.2:0/3974107179 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f5960034a20 con 0x7f5968109370 2024-03-11T18:32:48.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.443+0000 7f59711cc740 1 -- start start 2024-03-11T18:32:48.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 0x55f0c07e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07eb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c0752b30 con 0x55f0c07e3580 2024-03-11T18:32:48.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c0750060 con 0x55f0c07e3f70 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f59711cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c074b840 con 0x55f0c07e90e0 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40772/0 (socket says 172.21.15.2:40772) 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f5966a00640 1 -- 172.21.15.2:0/2828841638 learned_addr learned my addr 172.21.15.2:0/2828841638 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f5967400640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 0x55f0c07e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.444+0000 7f5966a00640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 -- 172.21.15.2:0/2828841638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 msgr2=0x55f0c07eb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07eb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 -- 172.21.15.2:0/2828841638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 msgr2=0x55f0c07e8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 0x55f0c07e8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 -- 172.21.15.2:0/2828841638 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0c07134c0 con 0x55f0c07e3580 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5967400640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 0x55f0c07e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5966a00640 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5950013990 tx=0x7f5950009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413b6ebbc4d93e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f595fe00640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 0x7f594c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f595cc00640 1 -- 172.21.15.2:0/2828841638 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950004050 con 0x55f0c07e3580 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f595cc00640 1 -- 172.21.15.2:0/2828841638 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f595000fe20 con 0x55f0c07e3580 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f595fe00640 1 -- 172.21.15.2:0/3974107179 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f59600efef0 con 0x7f5968109370 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f595cc00640 1 -- 172.21.15.2:0/2828841638 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59500054c0 con 0x55f0c07e3580 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.445+0000 7f5967400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 0x7f594c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 -- 172.21.15.2:0/2828841638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 msgr2=0x55f0c07e39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f5967400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 0x7f594c0a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55f0c07e4370 tx=0x7f5958009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5950013990 tx=0x7f5950009c00 comp rx=0 tx=0).stop 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 -- 172.21.15.2:0/2828841638 shutdown_connections 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07eb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3f70 0x55f0c07e8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 --2- 172.21.15.2:0/2828841638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3580 0x55f0c07e39a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 -- 172.21.15.2:0/2828841638 >> 172.21.15.2:0/2828841638 conn(0x55f0c07deaf0 msgr2=0x55f0c07deef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 -- 172.21.15.2:0/2828841638 shutdown_connections 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.446+0000 7f59711cc740 1 -- 172.21.15.2:0/2828841638 wait complete. 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 -- start start 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 Processor -- start 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e3f70 0x55f0c07ed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 0x55f0c07f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c06f0ea0 con 0x55f0c07e90e0 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c06f0b40 con 0x55f0c07e3580 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.447+0000 7f59711cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c0493f20 con 0x55f0c07e3f70 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f59711cc740 10 client.0 _ll_register_callbacks cb 0x55f0c0704950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45882/0 (socket says 172.21.15.2:45882) 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 -- 172.21.15.2:0/2261084648 learned_addr learned my addr 172.21.15.2:0/2261084648 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967e00640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 0x55f0c07f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5966a00640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e3f70 0x55f0c07ed9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 -- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e3f70 msgr2=0x55f0c07ed9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e3f70 0x55f0c07ed9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 -- 172.21.15.2:0/2261084648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 msgr2=0x55f0c07f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 0x55f0c07f49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967400640 1 -- 172.21.15.2:0/2261084648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59500135c0 con 0x55f0c07e3580 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.448+0000 7f5967e00640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 0x55f0c07f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5967400640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f595800fae0 tx=0x7f5958004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bd5286e1f4f1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5955600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958023070 con 0x55f0c07e3580 2024-03-11T18:32:48.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5958005020 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0c076fc40 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5958017070 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f59711cc740 10 client.25175 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f0c0749ec0 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.449+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f0c076fc40 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f595801d620 con 0x55f0c07e3580 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 10 client.25175 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 10 client.25175 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 20 client.25175 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c07f9a90 0x55f0c07f8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 4 client.25175 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 --> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f0c044b6c0 con 0x55f0c07f9a90 2024-03-11T18:32:48.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.450+0000 7f5966a00640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c07f9a90 0x55f0c07f8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.451+0000 7f5966a00640 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c07f9a90 0x55f0c07f8cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3375618c70d9d032 server_cookie=a735134c7e3a7251 in_seq=0 out_seq=0 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.451+0000 7f5955600640 10 client.25175 ms_handle_connect on v2:172.21.15.165:6838/1329073865 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f5955600640 1 -- 172.21.15.2:0/2261084648 <== mds.1 v2:172.21.15.165:6838/1329073865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f0c044b6c0 con 0x55f0c07f9a90 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f5955600640 10 client.25175 handle_command_reply: tid=0 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 client.25175 shutdown 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 msgr2=0x55f0c07ed460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f595800fae0 tx=0x7f5958004730 comp rx=0 tx=0).stop 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 shutdown_connections 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c07f9a90 0x55f0c07f8cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e90e0 0x55f0c07f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e3f70 0x55f0c07ed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 --2- 172.21.15.2:0/2261084648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07ed460 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f5955600640 0 client.25175 ms_handle_reset on v2:172.21.15.165:6838/1329073865 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 >> 172.21.15.2:0/2261084648 conn(0x55f0c07deaf0 msgr2=0x55f0c07dde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 shutdown_connections 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 1 -- 172.21.15.2:0/2261084648 wait complete. 2024-03-11T18:32:48.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.452+0000 7f59711cc740 20 client.25175 trim_cache size 0 max 16384 2024-03-11T18:32:48.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 Processor -- start 2024-03-11T18:32:48.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 -- start start 2024-03-11T18:32:48.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c07fa320 con 0x55f0c07e3f70 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c07f8ef0 con 0x55f0c07e3580 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f59711cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c044b6c0 con 0x55f0c07e90e0 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40796/0 (socket says 172.21.15.2:40796) 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f5966a00640 1 -- 172.21.15.2:0/1397206020 learned_addr learned my addr 172.21.15.2:0/1397206020 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f5967400640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.470+0000 7f5967e00640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 -- 172.21.15.2:0/1397206020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 msgr2=0x55f0c07e4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07e4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 -- 172.21.15.2:0/1397206020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 msgr2=0x55f0c07e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 -- 172.21.15.2:0/1397206020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0c06e68b0 con 0x55f0c07e3f70 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5967e00640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5967400640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:32:48.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5966a00640 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5950013960 tx=0x7f59500137e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6735741a307204 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5957e00640 1 -- 172.21.15.2:0/1397206020 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950026040 con 0x55f0c07e3f70 2024-03-11T18:32:48.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5957e00640 1 -- 172.21.15.2:0/1397206020 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f595002b340 con 0x55f0c07e3f70 2024-03-11T18:32:48.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.471+0000 7f5957e00640 1 -- 172.21.15.2:0/1397206020 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950005500 con 0x55f0c07e3f70 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 -- 172.21.15.2:0/1397206020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 msgr2=0x55f0c07f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5950013960 tx=0x7f59500137e0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 -- 172.21.15.2:0/1397206020 shutdown_connections 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f4970 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 --2- 172.21.15.2:0/1397206020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 -- 172.21.15.2:0/1397206020 >> 172.21.15.2:0/1397206020 conn(0x55f0c07deaf0 msgr2=0x55f0c0801f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 -- 172.21.15.2:0/1397206020 shutdown_connections 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.472+0000 7f59711cc740 1 -- 172.21.15.2:0/1397206020 wait complete. 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 -- start start 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 Processor -- start 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07f4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07f1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c07f1bb0 con 0x55f0c07e3f70 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c07f1d20 con 0x55f0c07e3580 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0c084f690 con 0x55f0c07e90e0 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f59711cc740 10 client.0 _ll_register_callbacks cb 0x7f5958009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.473+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5966a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40800/0 (socket says 172.21.15.2:40800) 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5966a00640 1 -- 172.21.15.2:0/722554230 learned_addr learned my addr 172.21.15.2:0/722554230 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 -- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 msgr2=0x55f0c07f1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07f1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 -- 172.21.15.2:0/722554230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 msgr2=0x55f0c07f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 -- 172.21.15.2:0/722554230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0c07134c0 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5966a00640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5967400640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07f4640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5958009530 tx=0x7f595801de70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faff13c914e02bb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5955600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958017070 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0c07fa460 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.474+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5958012040 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5958023040 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 10 client.25181 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f0c070c510 con 0x55f0c07e3580 2024-03-11T18:32:48.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f0c07fa460 con 0x55f0c07e3580 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5958005140 con 0x55f0c07e3580 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 10 client.25181 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 10 client.25181 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 20 client.25181 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c0854990 0x55f0c0850df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 4 client.25181 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.475+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 --> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f595000fe20 con 0x55f0c0854990 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f5966a00640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c0854990 0x55f0c0850df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f5966a00640 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c0854990 0x55f0c0850df0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=778d99e404afeac8 server_cookie=e1dab35f33053044 in_seq=0 out_seq=0 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f5955600640 10 client.25181 ms_handle_connect on v2:172.21.15.165:6838/1329073865 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f5955600640 1 -- 172.21.15.2:0/722554230 <== mds.1 v2:172.21.15.165:6838/1329073865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f595000fe20 con 0x55f0c0854990 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f5955600640 10 client.25181 handle_command_reply: tid=0 2024-03-11T18:32:48.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.476+0000 7f59711cc740 1 client.25181 shutdown 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 msgr2=0x55f0c07f4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07f4640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5958009530 tx=0x7f595801de70 comp rx=0 tx=0).stop 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 shutdown_connections 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x55f0c0854990 0x55f0c0850df0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f0c07e90e0 0x55f0c07f1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f0c07e3f70 0x55f0c07f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f5955600640 0 client.25181 ms_handle_reset on v2:172.21.15.165:6838/1329073865 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 --2- 172.21.15.2:0/722554230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0c07e3580 0x55f0c07f4640 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 >> 172.21.15.2:0/722554230 conn(0x55f0c07deaf0 msgr2=0x55f0c07e1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 shutdown_connections 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 1 -- 172.21.15.2:0/722554230 wait complete. 2024-03-11T18:32:48.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f59711cc740 20 client.25181 trim_cache size 0 max 16384 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24550, 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 109.05591895400001, 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:32:48.485 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:32:48.486 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 157.63651230799999 2024-03-11T18:32:48.486 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:32:48.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 msgr2=0x7f594c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.477+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 0x7f594c0a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55f0c07e4370 tx=0x7f5958009260 comp rx=0 tx=0).stop 2024-03-11T18:32:48.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 msgr2=0x7f59681a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5960033420 tx=0x7f5960033cb0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 shutdown_connections 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f594c0a1a30 0x7f594c0a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5968109370 0x7f59681a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59681089b0 0x7f596819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 --2- 172.21.15.2:0/3974107179 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59681077c0 0x7f596819d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 >> 172.21.15.2:0/3974107179 conn(0x7f5968076770 msgr2=0x7f596810bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 shutdown_connections 2024-03-11T18:32:48.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.478+0000 7f596e400640 1 -- 172.21.15.2:0/3974107179 wait complete. 2024-03-11T18:32:48.500 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 -- 172.21.15.2:0/233615192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef41033c0 msgr2=0x7f0ef41037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 --2- 172.21.15.2:0/233615192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef41033c0 0x7f0ef41037e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009900 tx=0x7f0ee802bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 -- 172.21.15.2:0/233615192 shutdown_connections 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 --2- 172.21.15.2:0/233615192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef4104f70 0x7f0ef410b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 --2- 172.21.15.2:0/233615192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41045b0 0x7f0ef4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 --2- 172.21.15.2:0/233615192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef41033c0 0x7f0ef41037e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 -- 172.21.15.2:0/233615192 >> 172.21.15.2:0/233615192 conn(0x7f0ef40fe930 msgr2=0x7f0ef4100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 -- 172.21.15.2:0/233615192 shutdown_connections 2024-03-11T18:32:48.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.706+0000 7f0efc800640 1 -- 172.21.15.2:0/233615192 wait complete. 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.707+0000 7f0efc800640 1 Processor -- start 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 -- start start 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41033c0 0x7f0ef4078f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 0x7f0ef4075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4076230 con 0x7f0ef41045b0 2024-03-11T18:32:48.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef40763a0 con 0x7f0ef4104f70 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efc800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4076560 con 0x7f0ef41033c0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40832/0 (socket says 172.21.15.2:40832) 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0ef9400640 1 -- 172.21.15.2:0/1332227244 learned_addr learned my addr 172.21.15.2:0/1332227244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0efa800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 0x7f0ef4075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.708+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41033c0 0x7f0ef4078f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 -- 172.21.15.2:0/1332227244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41033c0 msgr2=0x7f0ef4078f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41033c0 0x7f0ef4078f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 -- 172.21.15.2:0/1332227244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 msgr2=0x7f0ef4075aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 0x7f0ef4075aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 -- 172.21.15.2:0/1332227244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8009560 con 0x7f0ef41045b0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0efa800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 0x7f0ef4075aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0ef9400640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400e9f0 tx=0x7f0ee400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0eeea00640 1 -- 172.21.15.2:0/1332227244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4009720 con 0x7f0ef41045b0 2024-03-11T18:32:48.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.709+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef41a5980 con 0x7f0ef41045b0 2024-03-11T18:32:48.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.710+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef41a5e90 con 0x7f0ef41045b0 2024-03-11T18:32:48.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.710+0000 7f0eeea00640 1 -- 172.21.15.2:0/1332227244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee400cd80 con 0x7f0ef41045b0 2024-03-11T18:32:48.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.710+0000 7f0eeea00640 1 -- 172.21.15.2:0/1332227244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4010690 con 0x7f0ef41045b0 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.713+0000 7f0eff639740 1 Processor -- start 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.713+0000 7f0eff639740 1 -- start start 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.713+0000 7f0eeea00640 1 -- 172.21.15.2:0/1332227244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f0ee40046c0 con 0x7f0ef41045b0 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da610b30 con 0x55b2da6a70e0 2024-03-11T18:32:48.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da60e060 con 0x55b2da6a1f70 2024-03-11T18:32:48.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0eff639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da609840 con 0x55b2da6a1580 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0efa800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40844/0 (socket says 172.21.15.2:40844) 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.714+0000 7f0ef9400640 1 -- 172.21.15.2:0/2753073614 learned_addr learned my addr 172.21.15.2:0/2753073614 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 -- 172.21.15.2:0/2753073614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 msgr2=0x55b2da6a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 -- 172.21.15.2:0/2753073614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 msgr2=0x55b2da6a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 -- 172.21.15.2:0/2753073614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2da5d14c0 con 0x55b2da6a70e0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ef9400640 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d3a80 tx=0x7f0ee40d3b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f305b9c62dd4b09 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0eeea00640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 0x7f0edc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ee3000640 1 -- 172.21.15.2:0/2753073614 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4096970 con 0x55b2da6a70e0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ee3000640 1 -- 172.21.15.2:0/2753073614 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee40d2030 con 0x55b2da6a70e0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.715+0000 7f0ee3000640 1 -- 172.21.15.2:0/2753073614 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee407c890 con 0x55b2da6a70e0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eeea00640 1 -- 172.21.15.2:0/1332227244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f0ee4014070 con 0x7f0ef41045b0 2024-03-11T18:32:48.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0efa800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 0x7f0edc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 -- 172.21.15.2:0/2753073614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 msgr2=0x55b2da6a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d3a80 tx=0x7f0ee40d3b70 comp rx=0 tx=0).stop 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0efa800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 0x7f0edc0a3ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55b2da6a27d0 tx=0x7f0ef0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 -- 172.21.15.2:0/2753073614 shutdown_connections 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6a9570 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 --2- 172.21.15.2:0/2753073614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 -- 172.21.15.2:0/2753073614 >> 172.21.15.2:0/2753073614 conn(0x55b2da69caf0 msgr2=0x55b2da69cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.716+0000 7f0eff639740 1 -- 172.21.15.2:0/2753073614 shutdown_connections 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.717+0000 7f0eff639740 1 -- 172.21.15.2:0/2753073614 wait complete. 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.717+0000 7f0eff639740 1 -- start start 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.717+0000 7f0eff639740 1 Processor -- start 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6ab820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da5aeea0 con 0x55b2da6a70e0 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da5aeb40 con 0x55b2da6a1f70 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da351f20 con 0x55b2da6a1580 2024-03-11T18:32:48.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0eff639740 10 client.0 _ll_register_callbacks cb 0x55b2da5c2950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0efa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:33082/0 (socket says 172.21.15.2:33082) 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0ef9e00640 1 -- 172.21.15.2:0/4007845677 learned_addr learned my addr 172.21.15.2:0/4007845677 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.718+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 -- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 msgr2=0x55b2da6ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 -- 172.21.15.2:0/4007845677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 msgr2=0x55b2da6b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 -- 172.21.15.2:0/4007845677 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef41a5e90 con 0x55b2da6a1580 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000c00 tx=0x7f0ee80048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df6778428676821 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ed7400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:32:48.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8004ae0 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2da62dc40 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.719+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f0ee8046070 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ee8004250 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 10 client.25195 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b2da607ec0 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b2da62dc40 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ee803c040 con 0x55b2da6a1580 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 10 client.25195 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 10 client.25195 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 20 client.25195 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da6b7970 0x55b2da6b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 4 client.25195 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:32:48.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.720+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b2da3096c0 con 0x55b2da6b7970 2024-03-11T18:32:48.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.721+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da6b7970 0x55b2da6b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.721+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da6b7970 0x55b2da6b6ba0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe9f17c10360a12 server_cookie=39c90d9e1616883e in_seq=0 out_seq=0 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.721+0000 7f0ed7400640 10 client.25195 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0ed7400640 1 -- 172.21.15.2:0/4007845677 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b2da3096c0 con 0x55b2da6b7970 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0ed7400640 10 client.25195 handle_command_reply: tid=0 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 client.25195 shutdown 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 msgr2=0x55b2da6ab2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000c00 tx=0x7f0ee80048f0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 shutdown_connections 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da6b7970 0x55b2da6b6ba0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a70e0 0x55b2da6b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1f70 0x55b2da6ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 --2- 172.21.15.2:0/4007845677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6ab2e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0ed7400640 0 client.25195 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 >> 172.21.15.2:0/4007845677 conn(0x55b2da69caf0 msgr2=0x55b2da69be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.722+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 shutdown_connections 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.723+0000 7f0eff639740 1 -- 172.21.15.2:0/4007845677 wait complete. 2024-03-11T18:32:48.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.723+0000 7f0eff639740 20 client.25195 trim_cache size 0 max 16384 2024-03-11T18:32:48.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 Processor -- start 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 -- start start 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a70e0 0x55b2da6bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da6a99f0 con 0x55b2da6a1f70 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da3096c0 con 0x55b2da6a1580 2024-03-11T18:32:48.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.740+0000 7f0eff639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da62dc40 con 0x55b2da6a70e0 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45944/0 (socket says 172.21.15.2:45944) 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 -- 172.21.15.2:0/3792236068 learned_addr learned my addr 172.21.15.2:0/3792236068 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0efa800640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a70e0 0x55b2da6bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9400640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 -- 172.21.15.2:0/3792236068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a70e0 msgr2=0x55b2da6bfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a70e0 0x55b2da6bfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 -- 172.21.15.2:0/3792236068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 msgr2=0x55b2da6b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 -- 172.21.15.2:0/3792236068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2da5a48b0 con 0x55b2da6a1580 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9400640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.741+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=987dde0f7d0686f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0ee2600640 1 -- 172.21.15.2:0/3792236068 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80035f0 con 0x55b2da6a1580 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0ee2600640 1 -- 172.21.15.2:0/3792236068 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ee8044070 con 0x55b2da6a1580 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0ee2600640 1 -- 172.21.15.2:0/3792236068 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee804bbc0 con 0x55b2da6a1580 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 -- 172.21.15.2:0/3792236068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 msgr2=0x55b2da6a65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009850 tx=0x7f0ee802bc30 comp rx=0 tx=0).stop 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 -- 172.21.15.2:0/3792236068 shutdown_connections 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a70e0 0x55b2da6bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 --2- 172.21.15.2:0/3792236068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a1580 0x55b2da6a65d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 -- 172.21.15.2:0/3792236068 >> 172.21.15.2:0/3792236068 conn(0x55b2da69caf0 msgr2=0x55b2da69b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.742+0000 7f0eff639740 1 -- 172.21.15.2:0/3792236068 shutdown_connections 2024-03-11T18:32:48.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 -- 172.21.15.2:0/3792236068 wait complete. 2024-03-11T18:32:48.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 -- start start 2024-03-11T18:32:48.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 Processor -- start 2024-03-11T18:32:48.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.743+0000 7f0eff639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a70e0 0x55b2da6aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0eff639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da6af640 con 0x55b2da6a1f70 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0eff639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da6af7b0 con 0x55b2da6a70e0 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0eff639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2da6af970 con 0x55b2da6a1580 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0eff639740 10 client.0 _ll_register_callbacks cb 0x7f0ee8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40876/0 (socket says 172.21.15.2:40876) 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 -- 172.21.15.2:0/4200313436 learned_addr learned my addr 172.21.15.2:0/4200313436 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0efa800640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a70e0 0x55b2da6aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 -- 172.21.15.2:0/4200313436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 msgr2=0x55b2da6b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6b23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 -- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a70e0 msgr2=0x55b2da6aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a70e0 0x55b2da6aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.744+0000 7f0ef9400640 1 -- 172.21.15.2:0/4200313436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2da607ec0 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ef9400640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d5690 tx=0x7f0ee4085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad89443967c1209 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ed7400640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4083ea0 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f0ee40b8e40 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b2da6b7680 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ee40cb120 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0eff639740 10 client.15102 fetch_fsmap learned FSMap version 24 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b2da5ca510 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.745+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b2da6b7680 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ee401f060 con 0x55b2da6a1f70 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 10 client.15102 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:32:48.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 10 client.15102 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 20 client.15102 populate_metadata read hostname 'smithi002' 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da712250 0x55b2da70e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 4 client.15102 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0ee40d2030 con 0x55b2da712250 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.746+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da712250 0x55b2da70e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0ef9e00640 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da712250 0x55b2da70e6f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88b736e5460f236 server_cookie=58e615756f7eb640 in_seq=0 out_seq=0 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0ed7400640 10 client.15102 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0ed7400640 1 -- 172.21.15.2:0/4200313436 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7f0ee40d2030 con 0x55b2da712250 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0ed7400640 10 client.15102 handle_command_reply: tid=0 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 client.15102 shutdown 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 msgr2=0x55b2da6b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40d5690 tx=0x7f0ee4085040 comp rx=0 tx=0).stop 2024-03-11T18:32:48.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 shutdown_connections 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b2da712250 0x55b2da70e6f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2da6a70e0 0x55b2da6aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2da6a1f70 0x55b2da6b28f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0ed7400640 0 client.15102 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 --2- 172.21.15.2:0/4200313436 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2da6a1580 0x55b2da6b23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 >> 172.21.15.2:0/4200313436 conn(0x55b2da69caf0 msgr2=0x55b2da6bfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 shutdown_connections 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 1 -- 172.21.15.2:0/4200313436 wait complete. 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.747+0000 7f0eff639740 20 client.15102 trim_cache size 0 max 16384 2024-03-11T18:32:48.754 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24575, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 140.48116725400001, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 146.956993356 2024-03-11T18:32:48.755 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 msgr2=0x7f0edc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 0x7f0edc0a3ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55b2da6a27d0 tx=0x7f0ef0009500 comp rx=0 tx=0).stop 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 msgr2=0x7f0ef4079470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400e9f0 tx=0x7f0ee400edf0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 shutdown_connections 2024-03-11T18:32:48.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f0edc0a1a30 0x7f0edc0a3ef0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ef4104f70 0x7f0ef4075aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ef41045b0 0x7f0ef4079470 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 --2- 172.21.15.2:0/1332227244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ef41033c0 0x7f0ef4078f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 >> 172.21.15.2:0/1332227244 conn(0x7f0ef40fe930 msgr2=0x7f0ef41078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 shutdown_connections 2024-03-11T18:32:48.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:48.748+0000 7f0efc800640 1 -- 172.21.15.2:0/1332227244 wait complete. 2024-03-11T18:32:48.769 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T18:32:48.770 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T18:32:48.779 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:32:48.780 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T18:32:49.254 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:50.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:50 smithi002 ceph-mon[31328]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:50.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:50 smithi165 ceph-mon[36634]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:50 smithi023 ceph-mon[36406]: pgmap v395: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:51.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 -- 172.21.15.2:0/1959278593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668105060 msgr2=0x7fa668107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:51.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 --2- 172.21.15.2:0/1959278593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668105060 0x7fa668107450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa650009990 tx=0x7fa650009ca0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 -- 172.21.15.2:0/1959278593 shutdown_connections 2024-03-11T18:32:51.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 --2- 172.21.15.2:0/1959278593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668107990 0x7fa668109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 --2- 172.21.15.2:0/1959278593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668102730 0x7fa668104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 --2- 172.21.15.2:0/1959278593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668105060 0x7fa668107450 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 -- 172.21.15.2:0/1959278593 >> 172.21.15.2:0/1959278593 conn(0x7fa6680fc140 msgr2=0x7fa6680fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 -- 172.21.15.2:0/1959278593 shutdown_connections 2024-03-11T18:32:51.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.478+0000 7fa66e000700 1 -- 172.21.15.2:0/1959278593 wait complete. 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.479+0000 7fa66e000700 1 Processor -- start 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.479+0000 7fa66e000700 1 -- start start 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668107990 0x7fa668100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6681013a0 con 0x7fa668107990 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa668101510 con 0x7fa668105060 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa66e000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6681a1d80 con 0x7fa668102730 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa667e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668107990 0x7fa668100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa667400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:51.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa667400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:33116/0 (socket says 172.21.15.2:33116) 2024-03-11T18:32:51.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45970/0 (socket says 172.21.15.2:45970) 2024-03-11T18:32:51.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.480+0000 7fa667400700 1 -- 172.21.15.2:0/832016309 learned_addr learned my addr 172.21.15.2:0/832016309 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:51.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 -- 172.21.15.2:0/832016309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 msgr2=0x7fa668102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:51.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 -- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668107990 msgr2=0x7fa668100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668107990 0x7fa668100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 -- 172.21.15.2:0/832016309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa658009520 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa667400700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa65ea00700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098e0 tx=0x7fa650005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65001d070 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa650022470 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650017610 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.481+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6500095f0 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.482+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6681a2230 con 0x7fa668105060 2024-03-11T18:32:51.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.483+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6680fdcc0 con 0x7fa668105060 2024-03-11T18:32:51.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.485+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa6500225e0 con 0x7fa668105060 2024-03-11T18:32:51.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.486+0000 7fa664c00700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 0x7fa64c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:51.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.486+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fa6500ceaa0 con 0x7fa668105060 2024-03-11T18:32:51.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.486+0000 7fa667400700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 0x7fa64c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:51.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.487+0000 7fa667400700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 0x7fa64c0a3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa658009fd0 tx=0x7fa658009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:51.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.489+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa650097920 con 0x7fa668105060 2024-03-11T18:32:51.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.733+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa66802d8d0 con 0x7fa64c0a1ac0 2024-03-11T18:32:51.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.735+0000 7fa664c00700 1 -- 172.21.15.2:0/832016309 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fa66802d8d0 con 0x7fa64c0a1ac0 2024-03-11T18:32:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:Backend: cephadm 2024-03-11T18:32:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:Available: Yes 2024-03-11T18:32:51.738 INFO:teuthology.orchestra.run.smithi002.stdout:Paused: No 2024-03-11T18:32:51.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 msgr2=0x7fa64c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:51.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 0x7fa64c0a3f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa658009fd0 tx=0x7fa658009260 comp rx=0 tx=0).stop 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 msgr2=0x7fa6681006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098e0 tx=0x7fa650005f10 comp rx=0 tx=0).stop 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 shutdown_connections 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa64c0a1ac0 0x7fa64c0a3f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa668102730 0x7fa668102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa668107990 0x7fa668100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 --2- 172.21.15.2:0/832016309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa668105060 0x7fa6681006e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:51.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 >> 172.21.15.2:0/832016309 conn(0x7fa6680fc140 msgr2=0x7fa668103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:51.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 shutdown_connections 2024-03-11T18:32:51.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:51.738+0000 7fa66e000700 1 -- 172.21.15.2:0/832016309 wait complete. 2024-03-11T18:32:52.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:52 smithi002 ceph-mon[31328]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:52.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:52 smithi165 ceph-mon[36634]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:52 smithi023 ceph-mon[36406]: pgmap v396: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:32:52.573 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T18:32:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:53.189 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:53 smithi002 ceph-mon[31328]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:53.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:53 smithi165 ceph-mon[36634]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:53.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:53 smithi023 ceph-mon[36406]: from='client.25205 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:54.240 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:54 smithi002 ceph-mon[31328]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:54.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:54 smithi165 ceph-mon[36634]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:54.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:54 smithi023 ceph-mon[36406]: pgmap v397: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:32:55.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 -- 172.21.15.2:0/1551063522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8103850 msgr2=0x7f7de810a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:55.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 --2- 172.21.15.2:0/1551063522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8103850 0x7f7de810a100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7de0009f90 tx=0x7f7de000b330 comp rx=0 tx=0).stop 2024-03-11T18:32:55.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 -- 172.21.15.2:0/1551063522 shutdown_connections 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 --2- 172.21.15.2:0/1551063522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8103850 0x7f7de810a100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 --2- 172.21.15.2:0/1551063522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de81020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 --2- 172.21.15.2:0/1551063522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8102eb0 0x7f7de8103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 -- 172.21.15.2:0/1551063522 >> 172.21.15.2:0/1551063522 conn(0x7f7de80fd210 msgr2=0x7f7de80ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 -- 172.21.15.2:0/1551063522 shutdown_connections 2024-03-11T18:32:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.197+0000 7f7dee000700 1 -- 172.21.15.2:0/1551063522 wait complete. 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.198+0000 7f7dee000700 1 Processor -- start 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 -- start start 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de8199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 0x7f7de819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8103850 0x7f7de81a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81a1b00 con 0x7f7de8103850 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81a1c40 con 0x7f7de8101cb0 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7dee000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de81a1dd0 con 0x7f7de8102eb0 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7de7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de8199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7de7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de8199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45988/0 (socket says 172.21.15.2:45988) 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7de7400700 1 -- 172.21.15.2:0/4287527896 learned_addr learned my addr 172.21.15.2:0/4287527896 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7de6a00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 0x7f7de819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:55.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.199+0000 7f7de7e00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8103850 0x7f7de81a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:55.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 -- 172.21.15.2:0/4287527896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 msgr2=0x7f7de8199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:55.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de8199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 -- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8103850 msgr2=0x7f7de81a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:55.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8103850 0x7f7de81a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 -- 172.21.15.2:0/4287527896 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8009520 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.200+0000 7f7de6a00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 0x7f7de819a4b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400ebc0 tx=0x7f7dd400eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.201+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400cc90 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.201+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dd4004500 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.201+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4010430 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.201+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd40095f0 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.201+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de0009c40 con 0x7f7de8102eb0 2024-03-11T18:32:55.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.202+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de8068930 con 0x7f7de8102eb0 2024-03-11T18:32:55.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.204+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f7dd4003680 con 0x7f7de8102eb0 2024-03-11T18:32:55.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.205+0000 7f7ddfe00700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 0x7f7dcc0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:55.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.205+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f7dd4014070 con 0x7f7de8102eb0 2024-03-11T18:32:55.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.205+0000 7f7de7400700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 0x7f7dcc0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:55.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.206+0000 7f7de7400700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 0x7f7dcc0a4000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8005950 tx=0x7f7dd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:55.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.209+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7dd4096910 con 0x7f7de8102eb0 2024-03-11T18:32:55.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.447+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7de80755c0 con 0x7f7dcc0a1b40 2024-03-11T18:32:55.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.476+0000 7f7ddfe00700 1 -- 172.21.15.2:0/4287527896 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7f7de80755c0 con 0x7f7dcc0a1b40 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.a smithi002 running (2m) 35s ago 2m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 7139c4795fc3 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.b smithi023 running (2m) 85s ago 2m 25.4M - 19.0.0-1607-g94a53970 a6477a474d71 2bd0270a9338 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.c smithi165 running (2m) 85s ago 2m 15.9M - 19.0.0-1607-g94a53970 a6477a474d71 b3c934914db8 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.d smithi002 running (2m) 35s ago 2m 12.4M - 19.0.0-1607-g94a53970 a6477a474d71 9e62b891b5f6 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.e smithi023 running (2m) 85s ago 2m 15.8M - 19.0.0-1607-g94a53970 a6477a474d71 d5c02e53f71c 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.f smithi165 running (2m) 85s ago 2m 15.5M - 19.0.0-1607-g94a53970 a6477a474d71 b41286ffee33 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.g smithi002 running (2m) 35s ago 2m 12.3M - 19.0.0-1607-g94a53970 a6477a474d71 c1a9f97df2cf 2024-03-11T18:32:55.480 INFO:teuthology.orchestra.run.smithi002.stdout:mds.h smithi023 running (2m) 85s ago 2m 18.1M - 19.0.0-1607-g94a53970 a6477a474d71 829a15f74664 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mds.i smithi165 running (2m) 85s ago 2m 17.0M - 19.0.0-1607-g94a53970 a6477a474d71 e2e6bccfd5d6 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mds.j smithi002 running (3m) 35s ago 3m 12.5M - 19.0.0-1607-g94a53970 a6477a474d71 2d4a34e73cd7 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mds.k smithi023 running (3m) 85s ago 3m 15.3M - 19.0.0-1607-g94a53970 a6477a474d71 dbda0c850c9e 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mds.l smithi165 running (3m) 85s ago 3m 16.5M - 19.0.0-1607-g94a53970 a6477a474d71 2ac009db11b6 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi002 *:9283,8765 running (11m) 35s ago 11m 514M - 19.0.0-1607-g94a53970 a6477a474d71 b214df16bcc6 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi023 *:8443,8765 running (9m) 85s ago 9m 453M - 19.0.0-1607-g94a53970 a6477a474d71 574760bdc2eb 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.z smithi165 *:8443,8765 running (9m) 85s ago 9m 452M - 19.0.0-1607-g94a53970 a6477a474d71 fd233307c879 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (11m) 35s ago 11m 49.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 e64cf63ba829 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi023 running (9m) 85s ago 9m 40.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 500d4b783fd6 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi165 running (9m) 85s ago 9m 41.6M 2048M 19.0.0-1607-g94a53970 a6477a474d71 6ee334e5e5b0 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (8m) 35s ago 8m 44.8M 4273M 19.0.0-1607-g94a53970 a6477a474d71 6fc2dc659579 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi023 running (8m) 85s ago 8m 57.0M 4270M 19.0.0-1607-g94a53970 a6477a474d71 0053d058a159 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi165 running (7m) 85s ago 7m 54.6M 4270M 19.0.0-1607-g94a53970 a6477a474d71 64a77647f7ee 2024-03-11T18:32:55.481 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (7m) 35s ago 7m 51.1M 4273M 19.0.0-1607-g94a53970 a6477a474d71 7c8c276b655c 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi023 running (6m) 85s ago 6m 42.7M 4270M 19.0.0-1607-g94a53970 a6477a474d71 dded0ce3a69b 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi165 running (6m) 85s ago 6m 44.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 24b24c985734 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi002 running (5m) 35s ago 5m 42.1M 4273M 19.0.0-1607-g94a53970 a6477a474d71 23b81036e154 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi023 running (5m) 85s ago 5m 43.1M 4270M 19.0.0-1607-g94a53970 a6477a474d71 bc3dbf72b6bc 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 smithi165 running (4m) 85s ago 4m 43.5M 4270M 19.0.0-1607-g94a53970 a6477a474d71 2897ef1d0fbb 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 smithi002 running (4m) 35s ago 4m 48.7M 4273M 19.0.0-1607-g94a53970 a6477a474d71 d6eb8a798eed 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 smithi023 running (3m) 85s ago 3m 43.6M 4270M 19.0.0-1607-g94a53970 a6477a474d71 eb52e5386be3 2024-03-11T18:32:55.482 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 smithi165 running (3m) 85s ago 3m 41.7M 4270M 19.0.0-1607-g94a53970 a6477a474d71 d92d46cb35f1 2024-03-11T18:32:55.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 msgr2=0x7f7dcc0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 0x7f7dcc0a4000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8005950 tx=0x7f7dd8009260 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 msgr2=0x7f7de819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 0x7f7de819a4b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400ebc0 tx=0x7f7dd400eed0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 shutdown_connections 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7de8103850 0x7f7de81a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7de8101cb0 0x7f7de8199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f7dcc0a1b40 0x7f7dcc0a4000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 --2- 172.21.15.2:0/4287527896 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7de8102eb0 0x7f7de819a4b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 >> 172.21.15.2:0/4287527896 conn(0x7f7de80fd210 msgr2=0x7f7de8073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 shutdown_connections 2024-03-11T18:32:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:55.481+0000 7f7dee000700 1 -- 172.21.15.2:0/4287527896 wait complete. 2024-03-11T18:32:56.236 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:56 smithi002 ceph-mon[31328]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:56.313 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T18:32:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:56 smithi165 ceph-mon[36634]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:56.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:56 smithi023 ceph-mon[36406]: pgmap v398: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:32:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:32:57.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:57 smithi002 ceph-mon[31328]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:57.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:57 smithi165 ceph-mon[36634]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:57.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:57 smithi023 ceph-mon[36406]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:32:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:32:58 smithi165 ceph-mon[36634]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:32:58 smithi023 ceph-mon[36406]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:58.485 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:32:58 smithi002 ceph-mon[31328]: pgmap v399: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.048+0000 7f5abb400700 1 -- 172.21.15.2:0/1106858035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 msgr2=0x7f5ab410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.048+0000 7f5abb400700 1 --2- 172.21.15.2:0/1106858035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab410a0e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009fc0 tx=0x7f5aac00b330 comp rx=0 tx=0).stop 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 -- 172.21.15.2:0/1106858035 shutdown_connections 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 --2- 172.21.15.2:0/1106858035 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab41020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 --2- 172.21.15.2:0/1106858035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab410a0e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 --2- 172.21.15.2:0/1106858035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab41032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 -- 172.21.15.2:0/1106858035 >> 172.21.15.2:0/1106858035 conn(0x7f5ab40fd250 msgr2=0x7f5ab40ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:59.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 -- 172.21.15.2:0/1106858035 shutdown_connections 2024-03-11T18:32:59.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.049+0000 7f5abb400700 1 -- 172.21.15.2:0/1106858035 wait complete. 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 Processor -- start 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 -- start start 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab4199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab41a1b70 con 0x7f5ab4102e90 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab41a1cb0 con 0x7f5ab4103830 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.050+0000 7f5abb400700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab41a1e40 con 0x7f5ab4101c90 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab3e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab3e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36318/0 (socket says 172.21.15.2:36318) 2024-03-11T18:32:59.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab3e00700 1 -- 172.21.15.2:0/3111441032 learned_addr learned my addr 172.21.15.2:0/3111441032 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab4199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab9400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 -- 172.21.15.2:0/3111441032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 msgr2=0x7f5ab41a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab41a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 -- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 msgr2=0x7f5ab419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab8a00700 1 -- 172.21.15.2:0/3111441032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa00095f0 con 0x7f5ab4101c90 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab3e00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.051+0000 7f5ab9400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5ab8a00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab4199fe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800e9f0 tx=0x7f5aa800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8004220 con 0x7f5ab4101c90 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa8004380 con 0x7f5ab4101c90 2024-03-11T18:32:59.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8003c20 con 0x7f5ab4101c90 2024-03-11T18:32:59.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac009c40 con 0x7f5ab4101c90 2024-03-11T18:32:59.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.052+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab41a2440 con 0x7f5ab4101c90 2024-03-11T18:32:59.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.053+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4068930 con 0x7f5ab4101c90 2024-03-11T18:32:59.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.056+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f5aa800c7c0 con 0x7f5ab4101c90 2024-03-11T18:32:59.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.057+0000 7f5ab1600700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 0x7f5a9c0a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:32:59.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.057+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f5aa8014070 con 0x7f5ab4101c90 2024-03-11T18:32:59.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.057+0000 7f5ab3e00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 0x7f5a9c0a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:32:59.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.058+0000 7f5ab3e00700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 0x7f5a9c0a4000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0005d10 tx=0x7f5aa0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:32:59.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.059+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5aa80968e0 con 0x7f5ab4101c90 2024-03-11T18:32:59.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.306+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ab40755c0 con 0x7f5a9c0a1b40 2024-03-11T18:32:59.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.313+0000 7f5ab1600700 1 -- 172.21.15.2:0/3111441032 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f5ab40755c0 con 0x7f5a9c0a1b40 2024-03-11T18:32:59.314 INFO:teuthology.orchestra.run.smithi002.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T18:32:59.315 INFO:teuthology.orchestra.run.smithi002.stdout:mds.all 12/12 89s ago 3m smithi002=a;smithi002=d;smithi002=g;smithi002=j;smithi023=b;smithi023=e;smithi023=h;smithi023=k;smithi165=c;smithi165=f;smithi165=i;smithi165=l;count:12 2024-03-11T18:32:59.315 INFO:teuthology.orchestra.run.smithi002.stdout:mgr 3/3 89s ago 9m smithi002=x;smithi023=y;smithi165=z;count:3 2024-03-11T18:32:59.315 INFO:teuthology.orchestra.run.smithi002.stdout:mon 3/3 89s ago 9m smithi002:172.21.15.2=a;smithi023:172.21.15.23=b;smithi165:172.21.15.165=c;count:3 2024-03-11T18:32:59.315 INFO:teuthology.orchestra.run.smithi002.stdout:osd 12 89s ago - 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.315+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 msgr2=0x7f5a9c0a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 0x7f5a9c0a4000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0005d10 tx=0x7f5aa0009310 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 msgr2=0x7f5ab4199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab4199fe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800e9f0 tx=0x7f5aa800ed00 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 shutdown_connections 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f5a9c0a1b40 0x7f5a9c0a4000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ab4101c90 0x7f5ab4199fe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ab4103830 0x7f5ab41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 --2- 172.21.15.2:0/3111441032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ab4102e90 0x7f5ab419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 >> 172.21.15.2:0/3111441032 conn(0x7f5ab40fd250 msgr2=0x7f5ab4073ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:32:59.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 shutdown_connections 2024-03-11T18:32:59.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:32:59.316+0000 7f5abb400700 1 -- 172.21.15.2:0/3111441032 wait complete. 2024-03-11T18:33:00.152 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T18:33:00.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:00 smithi165 ceph-mon[36634]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:00 smithi023 ceph-mon[36406]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:00.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:00 smithi002 ceph-mon[31328]: pgmap v400: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:00.661 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:33:01.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:01 smithi165 ceph-mon[36634]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:01.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:01 smithi023 ceph-mon[36406]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:01 smithi002 ceph-mon[31328]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:02.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:02 smithi165 ceph-mon[36634]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:02.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:02 smithi023 ceph-mon[36406]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:02.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:02 smithi002 ceph-mon[31328]: pgmap v401: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 -- 172.21.15.2:0/3783731833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840102740 msgr2=0x7fe840108ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 --2- 172.21.15.2:0/3783731833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840102740 0x7fe840108ff0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe83800b0d0 tx=0x7fe83800b3e0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 -- 172.21.15.2:0/3783731833 shutdown_connections 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 --2- 172.21.15.2:0/3783731833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840102740 0x7fe840108ff0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 --2- 172.21.15.2:0/3783731833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840100ba0 0x7fe840100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 --2- 172.21.15.2:0/3783731833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe840102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 -- 172.21.15.2:0/3783731833 >> 172.21.15.2:0/3783731833 conn(0x7fe8400fc140 msgr2=0x7fe8400fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 -- 172.21.15.2:0/3783731833 shutdown_connections 2024-03-11T18:33:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.067+0000 7fe845a00700 1 -- 172.21.15.2:0/3783731833 wait complete. 2024-03-11T18:33:03.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.068+0000 7fe845a00700 1 Processor -- start 2024-03-11T18:33:03.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.068+0000 7fe845a00700 1 -- start start 2024-03-11T18:33:03.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe84019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840102740 0x7fe8401a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8401a1b60 con 0x7fe840100ba0 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8401a1ca0 con 0x7fe840102740 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe845a00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8401a1e30 con 0x7fe840101da0 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe83ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe83ea00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36324/0 (socket says 172.21.15.2:36324) 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe83ea00700 1 -- 172.21.15.2:0/3712410992 learned_addr learned my addr 172.21.15.2:0/3712410992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.069+0000 7fe837e00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe84019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83f400700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840102740 0x7fe8401a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 -- 172.21.15.2:0/3712410992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 msgr2=0x7fe84019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe84019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 -- 172.21.15.2:0/3712410992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840102740 msgr2=0x7fe8401a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840102740 0x7fe8401a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 -- 172.21.15.2:0/3712410992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838009d00 con 0x7fe840100ba0 2024-03-11T18:33:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe837e00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe84019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.070+0000 7fe83ea00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe83000e9a0 tx=0x7fe83000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.071+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830009790 con 0x7fe840100ba0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.071+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe83000ccf0 con 0x7fe840100ba0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.071+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8401a2110 con 0x7fe840100ba0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.071+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830003d20 con 0x7fe840100ba0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.071+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8401a2660 con 0x7fe840100ba0 2024-03-11T18:33:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.072+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe840068930 con 0x7fe840100ba0 2024-03-11T18:33:03.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.074+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fe830004390 con 0x7fe840100ba0 2024-03-11T18:33:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.075+0000 7fe837400700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 0x7fe8240a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.075+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fe830014070 con 0x7fe840100ba0 2024-03-11T18:33:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.076+0000 7fe837e00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 0x7fe8240a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:03.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.076+0000 7fe837e00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 0x7fe8240a4000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe828011490 tx=0x7fe828009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:03.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.080+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe830096770 con 0x7fe840100ba0 2024-03-11T18:33:03.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.313+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe84010b060 con 0x7fe8240a1b40 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.316+0000 7fe837400700 1 -- 172.21.15.2:0/3712410992 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fe84010b060 con 0x7fe8240a1b40 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stdout:HOST ADDR LABELS STATUS 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 172.21.15.2 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stdout:smithi023 172.21.15.23 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 172.21.15.165 2024-03-11T18:33:03.318 INFO:teuthology.orchestra.run.smithi002.stdout:3 hosts in cluster 2024-03-11T18:33:03.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 msgr2=0x7fe8240a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:03.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 0x7fe8240a4000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe828011490 tx=0x7fe828009260 comp rx=0 tx=0).stop 2024-03-11T18:33:03.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 msgr2=0x7fe840199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:03.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe83000e9a0 tx=0x7fe83000ecb0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 shutdown_connections 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe840102740 0x7fe8401a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe840100ba0 0x7fe840199fd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fe8240a1b40 0x7fe8240a4000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 --2- 172.21.15.2:0/3712410992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe840101da0 0x7fe84019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.319+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 >> 172.21.15.2:0/3712410992 conn(0x7fe8400fc140 msgr2=0x7fe840109940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.320+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 shutdown_connections 2024-03-11T18:33:03.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:03.320+0000 7fe845a00700 1 -- 172.21.15.2:0/3712410992 wait complete. 2024-03-11T18:33:04.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T18:33:04.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:04 smithi165 ceph-mon[36634]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:04.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:04 smithi023 ceph-mon[36406]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:04 smithi002 ceph-mon[31328]: pgmap v402: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:04.666 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:33:05.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:05 smithi165 ceph-mon[36634]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:05.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:05 smithi023 ceph-mon[36406]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:05.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:05 smithi002 ceph-mon[31328]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:06.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:06 smithi165 ceph-mon[36634]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:06 smithi023 ceph-mon[36406]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:06.442 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:06 smithi002 ceph-mon[31328]: pgmap v403: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:06.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.967+0000 7fd93d600700 1 -- 172.21.15.2:0/1332685648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938102650 msgr2=0x7fd938104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.967+0000 7fd93d600700 1 --2- 172.21.15.2:0/1332685648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938102650 0x7fd938104a40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd928009990 tx=0x7fd928009ca0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 -- 172.21.15.2:0/1332685648 shutdown_connections 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 --2- 172.21.15.2:0/1332685648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9381078b0 0x7fd938109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 --2- 172.21.15.2:0/1332685648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938102650 0x7fd938104a40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 --2- 172.21.15.2:0/1332685648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 -- 172.21.15.2:0/1332685648 >> 172.21.15.2:0/1332685648 conn(0x7fd9380fc060 msgr2=0x7fd9380fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 -- 172.21.15.2:0/1332685648 shutdown_connections 2024-03-11T18:33:06.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.968+0000 7fd93d600700 1 -- 172.21.15.2:0/1332685648 wait complete. 2024-03-11T18:33:06.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.969+0000 7fd93d600700 1 Processor -- start 2024-03-11T18:33:06.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.969+0000 7fd93d600700 1 -- start start 2024-03-11T18:33:06.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd938102650 0x7fd938101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:06.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:06.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:06.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381012c0 con 0x7fd938102650 2024-03-11T18:33:06.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938101430 con 0x7fd9381078b0 2024-03-11T18:33:06.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93d600700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381a1b60 con 0x7fd938104f80 2024-03-11T18:33:06.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93cc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:06.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd937e00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd938102650 0x7fd938101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:06.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93cc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:36596/0 (socket says 172.21.15.2:36596) 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.970+0000 7fd93cc00700 1 -- 172.21.15.2:0/399623361 learned_addr learned my addr 172.21.15.2:0/399623361 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd937400700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 -- 172.21.15.2:0/399623361 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 msgr2=0x7fd938100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 -- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd938102650 msgr2=0x7fd938101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd938102650 0x7fd938101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 -- 172.21.15.2:0/399623361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930009c40 con 0x7fd9381078b0 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd937400700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.971+0000 7fd93cc00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd93000c110 tx=0x7fd93000c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.972+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930017070 con 0x7fd9381078b0 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.972+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9280095f0 con 0x7fd9381078b0 2024-03-11T18:33:06.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.972+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd93000f040 con 0x7fd9381078b0 2024-03-11T18:33:06.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.972+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93000e050 con 0x7fd9381078b0 2024-03-11T18:33:06.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.972+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381a2100 con 0x7fd9381078b0 2024-03-11T18:33:06.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.973+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9380fdbe0 con 0x7fd9381078b0 2024-03-11T18:33:06.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.975+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fd93000ca20 con 0x7fd9381078b0 2024-03-11T18:33:06.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.976+0000 7fd934c00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 0x7fd91c0a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:06.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.977+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fd9300cd060 con 0x7fd9381078b0 2024-03-11T18:33:06.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.977+0000 7fd937e00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 0x7fd91c0a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:06.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.977+0000 7fd937e00700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 0x7fd91c0a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b650 tx=0x7fd928005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:06.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:06.981+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd930100d20 con 0x7fd9381078b0 2024-03-11T18:33:07.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.215+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd93802d870 con 0x7fd91c0a1ac0 2024-03-11T18:33:07.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.219+0000 7fd934c00700 1 -- 172.21.15.2:0/399623361 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fd93802d870 con 0x7fd91c0a1ac0 2024-03-11T18:33:07.221 INFO:teuthology.orchestra.run.smithi002.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T18:33:07.221 INFO:teuthology.orchestra.run.smithi002.stdout:smithi002 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300EV480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-11T18:33:07.222 INFO:teuthology.orchestra.run.smithi002.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 4m ago Has a FileSystem, LVM detected 2024-03-11T18:33:07.222 INFO:teuthology.orchestra.run.smithi002.stdout:smithi165 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CE400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-11T18:33:07.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.222+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 msgr2=0x7fd91c0a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:07.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.222+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 0x7fd91c0a3f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b650 tx=0x7fd928005fb0 comp rx=0 tx=0).stop 2024-03-11T18:33:07.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.222+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 msgr2=0x7fd938100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:07.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.222+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd93000c110 tx=0x7fd93000c420 comp rx=0 tx=0).stop 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 shutdown_connections 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9381078b0 0x7fd938100b40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd938102650 0x7fd938101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fd91c0a1ac0 0x7fd91c0a3f80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 --2- 172.21.15.2:0/399623361 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd938104f80 0x7fd938100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 >> 172.21.15.2:0/399623361 conn(0x7fd9380fc060 msgr2=0x7fd938102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 shutdown_connections 2024-03-11T18:33:07.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:07.223+0000 7fd93d600700 1 -- 172.21.15.2:0/399623361 wait complete. 2024-03-11T18:33:08.217 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T18:33:08.226 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:33:08.226 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-11T18:33:08.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:08 smithi023 ceph-mon[36406]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:08.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:08 smithi165 ceph-mon[36634]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:08 smithi002 ceph-mon[31328]: pgmap v404: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:08.697 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:33:09.400 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:09 smithi002 ceph-mon[31328]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:09.400 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:09 smithi002 ceph-mon[31328]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:09.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:09 smithi165 ceph-mon[36634]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:09.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:09 smithi165 ceph-mon[36634]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:09 smithi023 ceph-mon[36406]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:09.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:09 smithi023 ceph-mon[36406]: pgmap v405: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 -- 172.21.15.2:0/3030565254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 msgr2=0x7fa1fc110740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 --2- 172.21.15.2:0/3030565254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc110740 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009fc0 tx=0x7fa1f400b330 comp rx=0 tx=0).stop 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 -- 172.21.15.2:0/3030565254 shutdown_connections 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 --2- 172.21.15.2:0/3030565254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc110740 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 --2- 172.21.15.2:0/3030565254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:10.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 --2- 172.21.15.2:0/3030565254 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 0x7fa1fc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:10.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 -- 172.21.15.2:0/3030565254 >> 172.21.15.2:0/3030565254 conn(0x7fa1fc0772c0 msgr2=0x7fa1fc079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:10.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 -- 172.21.15.2:0/3030565254 shutdown_connections 2024-03-11T18:33:10.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.959+0000 7fa203000700 1 -- 172.21.15.2:0/3030565254 wait complete. 2024-03-11T18:33:10.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.960+0000 7fa203000700 1 Processor -- start 2024-03-11T18:33:10.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 -- start start 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 0x7fa1fc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc1a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1a5eb0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1a5ff0 con 0x7fa1fc109e90 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa203000700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1a6180 con 0x7fa1fc1094f0 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa1fbe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa1fbe00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49894/0 (socket says 172.21.15.2:49894) 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa1fbe00700 1 -- 172.21.15.2:0/4239745813 learned_addr learned my addr 172.21.15.2:0/4239745813 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa1fb400700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 0x7fa1fc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.961+0000 7fa201000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc1a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:10.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 -- 172.21.15.2:0/4239745813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 msgr2=0x7fa1fc19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 0x7fa1fc19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 -- 172.21.15.2:0/4239745813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 msgr2=0x7fa1fc1a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc1a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 -- 172.21.15.2:0/4239745813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4009c40 con 0x7fa1fc1082f0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa201000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc1a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1fbe00700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00e9a0 tx=0x7fa1ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec009790 con 0x7fa1fc1082f0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec00cca0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc1a6460 con 0x7fa1fc1082f0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.962+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec003cb0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.963+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc1a69b0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.964+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc10f5e0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.966+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa1ec00c7c0 con 0x7fa1fc1082f0 2024-03-11T18:33:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.967+0000 7fa1f8c00700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 0x7fa1e00a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.968+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fa1ec014070 con 0x7fa1fc1082f0 2024-03-11T18:33:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.968+0000 7fa1fb400700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 0x7fa1e00a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:10.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.968+0000 7fa1fb400700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 0x7fa1e00a4000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005fd0 tx=0x7fa1e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:10.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:10.973+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1ec096750 con 0x7fa1fc1082f0 2024-03-11T18:33:11.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.242+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa1fc02d8a0 con 0x7fa1fc1082f0 2024-03-11T18:33:11.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.243+0000 7fa1f8c00700 1 -- 172.21.15.2:0/4239745813 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2935 (secure 0 0 0) 0x7fa1ec099e10 con 0x7fa1fc1082f0 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:e24 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:legacy client fscid: 1 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:Filesystem 'cephfs' (1) 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:fs_name cephfs 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 24 2024-03-11T18:33:11.246 INFO:teuthology.orchestra.run.smithi002.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-03-11T18:30:27.790111+0000 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-03-11T18:31:01.455980+0000 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:tableserver 0 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:root 0 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:session_timeout 300 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:session_autoclose 300 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:max_file_size 1099511627776 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:max_xattr_size 65536 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:required_client_features {} 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure 0 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:last_failure_osd_epoch 0 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:max_mds 3 2024-03-11T18:33:11.247 INFO:teuthology.orchestra.run.smithi002.stdout:in 0,1,2 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:up {0=24575,1=24550,2=24521} 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:failed 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:damaged 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:stopped 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:data_pools [3,4] 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:metadata_pool 2 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:inline_data disabled 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:balancer 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:bal_rank_mask -1 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:standby_count_wanted 0 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b{0:24575} state up:active seq 3 addr [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c{0:24520} state up:standby-replay seq 1 addr [v2:172.21.15.165:6836/3141279332,v1:172.21.15.165:6837/3141279332] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.i{1:24550} state up:active seq 14 addr [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.248 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.f{1:24508} state up:standby-replay seq 1 addr [v2:172.21.15.165:6834/2758205355,v1:172.21.15.165:6835/2758205355] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.h{2:24521} state up:active seq 15 addr [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.e{2:24494} state up:standby-replay seq 1 addr [v2:172.21.15.23:6834/1605486605,v1:172.21.15.23:6835/1605486605] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:Standby daemons: 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.j{-1:14394} state up:standby seq 1 addr [v2:172.21.15.2:6834/2188580308,v1:172.21.15.2:6835/2188580308] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.g{-1:14460} state up:standby seq 1 addr [v2:172.21.15.2:6838/3374917731,v1:172.21.15.2:6839/3374917731] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a{-1:14478} state up:standby seq 1 addr [v2:172.21.15.2:6840/2401819833,v1:172.21.15.2:6841/2401819833] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.165:6832/4285430616,v1:172.21.15.165:6833/4285430616] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.k{-1:24446} state up:standby seq 1 addr [v2:172.21.15.23:6832/2120271757,v1:172.21.15.23:6833/2120271757] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d{-1:24490} state up:standby seq 1 addr [v2:172.21.15.2:6836/2729524869,v1:172.21.15.2:6837/2729524869] compat {c=[1],r=[1],i=[fff]}] 2024-03-11T18:33:11.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 msgr2=0x7fa1e00a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:11.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 0x7fa1e00a4000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa1e8005fd0 tx=0x7fa1e8009310 comp rx=0 tx=0).stop 2024-03-11T18:33:11.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 msgr2=0x7fa1fc19e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:11.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00e9a0 tx=0x7fa1ec00ecb0 comp rx=0 tx=0).stop 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 shutdown_connections 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1fc109e90 0x7fa1fc1a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1fc1082f0 0x7fa1fc19e320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1e00a1b40 0x7fa1e00a4000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 --2- 172.21.15.2:0/4239745813 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1fc1094f0 0x7fa1fc19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.248+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 >> 172.21.15.2:0/4239745813 conn(0x7fa1fc0772c0 msgr2=0x7fa1fc10c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.249+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 shutdown_connections 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:11.249+0000 7fa203000700 1 -- 172.21.15.2:0/4239745813 wait complete. 2024-03-11T18:33:11.251 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:33:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:11 smithi002 ceph-mon[31328]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:11.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:11 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4239745813' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T18:33:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:11 smithi165 ceph-mon[36634]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:11.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:11 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4239745813' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T18:33:11.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:11 smithi023 ceph-mon[36406]: pgmap v406: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:11.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:11 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4239745813' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T18:33:12.006 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-11T18:33:12.491 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:33:13.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:13 smithi023 ceph-mon[36406]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:13.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:13 smithi165 ceph-mon[36634]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:13.705 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:13 smithi002 ceph-mon[31328]: pgmap v407: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:14.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 -- 172.21.15.2:0/1025157288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb281081e0 msgr2=0x7ffb28108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:14.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 --2- 172.21.15.2:0/1025157288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb281081e0 0x7ffb28108600 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c0099c0 tx=0x7ffb1c009cd0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 -- 172.21.15.2:0/1025157288 shutdown_connections 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 --2- 172.21.15.2:0/1025157288 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb28109d80 0x7ffb28110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 --2- 172.21.15.2:0/1025157288 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb281081e0 0x7ffb28108600 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 --2- 172.21.15.2:0/1025157288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 0x7ffb28109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 -- 172.21.15.2:0/1025157288 >> 172.21.15.2:0/1025157288 conn(0x7ffb280771b0 msgr2=0x7ffb280795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 -- 172.21.15.2:0/1025157288 shutdown_connections 2024-03-11T18:33:14.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.779+0000 7ffb2de00700 1 -- 172.21.15.2:0/1025157288 wait complete. 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 Processor -- start 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 -- start start 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 0x7ffb2819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb28109d80 0x7ffb281a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb281a5d50 con 0x7ffb28109d80 2024-03-11T18:33:14.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb281a5e90 con 0x7ffb281093e0 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2de00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb281a6020 con 0x7ffb281081e0 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.781+0000 7ffb2ca00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2d400700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb28109d80 0x7ffb281a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48206/0 (socket says 172.21.15.2:48206) 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 -- 172.21.15.2:0/2982029790 learned_addr learned my addr 172.21.15.2:0/2982029790 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:14.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb27e00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 0x7ffb2819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 -- 172.21.15.2:0/2982029790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 msgr2=0x7ffb2819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 0x7ffb2819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 -- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb28109d80 msgr2=0x7ffb281a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb28109d80 0x7ffb281a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 -- 172.21.15.2:0/2982029790 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0095f0 con 0x7ffb281081e0 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.782+0000 7ffb2ca00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009990 tx=0x7ffb1c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.783+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c01d070 con 0x7ffb281081e0 2024-03-11T18:33:14.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.783+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb281a62a0 con 0x7ffb281081e0 2024-03-11T18:33:14.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.783+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb281a6710 con 0x7ffb281081e0 2024-03-11T18:33:14.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.783+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb1c022470 con 0x7ffb281081e0 2024-03-11T18:33:14.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.783+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c017650 con 0x7ffb281081e0 2024-03-11T18:33:14.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.784+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2810f5a0 con 0x7ffb281081e0 2024-03-11T18:33:14.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.786+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7ffb1c0225e0 con 0x7ffb281081e0 2024-03-11T18:33:14.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.787+0000 7ffb25600700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 0x7ffb100a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:14.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.787+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ffb1c0ce6f0 con 0x7ffb281081e0 2024-03-11T18:33:14.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.788+0000 7ffb27e00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 0x7ffb100a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:14.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.788+0000 7ffb27e00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 0x7ffb100a4000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009c60 tx=0x7ffb14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:14.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:14.790+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffb1c097710 con 0x7ffb281081e0 2024-03-11T18:33:15.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.020+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7ffb28113d00 con 0x7ffb281081e0 2024-03-11T18:33:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.021+0000 7ffb25600700 1 -- 172.21.15.2:0/2982029790 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) v1 ==== 56+0+5649 (secure 0 0 0) 0x7ffb1c09add0 con 0x7ffb281081e0 2024-03-11T18:33:15.024 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 95 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:fsid 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-03-11T18:21:20.287842+0000 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:modified 2024-03-11T18:32:29.901057+0000 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:crush_version 27 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:full_ratio 0.95 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:backfillfull_ratio 0.9 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:nearfull_ratio 0.85 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:require_min_compat_client luminous 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:min_compat_client luminous 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:require_osd_release squid 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:stretch_mode_enabled false 2024-03-11T18:33:15.025 INFO:teuthology.orchestra.run.smithi002.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:max_osd 12 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6802/1098127519,v1:172.21.15.2:6803/1098127519] [v2:172.21.15.2:6804/1098127519,v1:172.21.15.2:6805/1098127519] exists,up a773fb9c-07bb-4b3e-94d4-8c382a21561e 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6800/1383496716,v1:172.21.15.23:6801/1383496716] [v2:172.21.15.23:6802/1383496716,v1:172.21.15.23:6803/1383496716] exists,up ec6ef905-1b64-44d7-bbd0-d52bc91b22df 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6800/758382799,v1:172.21.15.165:6801/758382799] [v2:172.21.15.165:6802/758382799,v1:172.21.15.165:6803/758382799] exists,up fa7a2e4d-c63a-479a-b306-c3c983dea728 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 up in weight 1 up_from 27 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6810/2581962705,v1:172.21.15.2:6811/2581962705] [v2:172.21.15.2:6812/2581962705,v1:172.21.15.2:6813/2581962705] exists,up 332dc7bc-07d1-4637-b31b-8c2e2171d1da 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 up in weight 1 up_from 35 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6808/2973022110,v1:172.21.15.23:6809/2973022110] [v2:172.21.15.23:6810/2973022110,v1:172.21.15.23:6811/2973022110] exists,up 1fa522c2-c663-4c43-b022-ce54054575ab 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 up in weight 1 up_from 40 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6808/1264754008,v1:172.21.15.165:6809/1264754008] [v2:172.21.15.165:6810/1264754008,v1:172.21.15.165:6811/1264754008] exists,up c0f4c30d-2ccb-4ed3-8562-a935e88af3b9 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 up in weight 1 up_from 48 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6818/2663871562,v1:172.21.15.2:6819/2663871562] [v2:172.21.15.2:6820/2663871562,v1:172.21.15.2:6821/2663871562] exists,up d1f9bcec-338a-44b6-bddb-920dac2a46e0 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 up in weight 1 up_from 53 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6816/2299093743,v1:172.21.15.23:6817/2299093743] [v2:172.21.15.23:6818/2299093743,v1:172.21.15.23:6819/2299093743] exists,up 787a342e-4f3a-47f0-9ff8-cf7e80400e7d 2024-03-11T18:33:15.026 INFO:teuthology.orchestra.run.smithi002.stdout:osd.8 up in weight 1 up_from 61 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6816/2939503461,v1:172.21.15.165:6817/2939503461] [v2:172.21.15.165:6818/2939503461,v1:172.21.15.165:6819/2939503461] exists,up 4e50d064-174a-482a-8643-898a23138c36 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:osd.9 up in weight 1 up_from 69 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.2:6826/1412624036,v1:172.21.15.2:6827/1412624036] [v2:172.21.15.2:6828/1412624036,v1:172.21.15.2:6829/1412624036] exists,up 1ca1d014-c271-49e5-9672-4ff1457beb85 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6824/1131591289,v1:172.21.15.23:6825/1131591289] [v2:172.21.15.23:6826/1131591289,v1:172.21.15.23:6827/1131591289] exists,up f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:osd.11 up in weight 1 up_from 82 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.165:6824/1832747584,v1:172.21.15.165:6825/1832747584] [v2:172.21.15.165:6826/1832747584,v1:172.21.15.165:6827/1832747584] exists,up 50c89611-028f-40d8-a5b0-d36c76cd8800 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.0 [8,4] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.b [2,1] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.10 [8,11] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.1b [2,4] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 2.1f [7,11] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.c [3,9] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.1b [4,8,5,9] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.21 [4,6] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 3.24 [4,9] 2024-03-11T18:33:15.027 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.5 [5,10] 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.b [9,8] 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.e [9,8] 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:pg_upmap_items 4.13 [9,8] 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/2981412802 expires 2024-03-12T18:22:28.050188+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/2981412802 expires 2024-03-12T18:22:28.050188+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/1969024355 expires 2024-03-12T18:22:28.050188+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3822870648 expires 2024-03-12T18:21:53.065562+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/247107605 expires 2024-03-12T18:21:53.065562+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3920276264 expires 2024-03-12T18:22:28.050188+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3359009707 expires 2024-03-12T18:21:53.065562+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6800/1779941333 expires 2024-03-12T18:21:53.065562+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:0/3303291273 expires 2024-03-12T18:22:28.050188+0000 2024-03-11T18:33:15.028 INFO:teuthology.orchestra.run.smithi002.stdout:blocklist 172.21.15.2:6801/1779941333 expires 2024-03-12T18:21:53.065562+0000 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 msgr2=0x7ffb100a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 0x7ffb100a4000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009c60 tx=0x7ffb14009260 comp rx=0 tx=0).stop 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 msgr2=0x7ffb2819e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009990 tx=0x7ffb1c00f690 comp rx=0 tx=0).stop 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 shutdown_connections 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ffb100a1b40 0x7ffb100a4000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:15.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb281081e0 0x7ffb2819e210 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:15.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb28109d80 0x7ffb281a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:15.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 --2- 172.21.15.2:0/2982029790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb281093e0 0x7ffb2819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:15.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 >> 172.21.15.2:0/2982029790 conn(0x7ffb280771b0 msgr2=0x7ffb2810c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:15.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 shutdown_connections 2024-03-11T18:33:15.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:15.026+0000 7ffb2de00700 1 -- 172.21.15.2:0/2982029790 wait complete. 2024-03-11T18:33:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:15 smithi165 ceph-mon[36634]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:15.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:15 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2982029790' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T18:33:15.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:15 smithi023 ceph-mon[36406]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:15.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:15 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2982029790' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T18:33:15.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:15 smithi002 ceph-mon[31328]: pgmap v408: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:15.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:15 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2982029790' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T18:33:15.810 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-11T18:33:15.823 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.062+0000 7f84ce400640 1 -- 172.21.15.2:0/4001295902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104e50 msgr2=0x7f84c810b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.062+0000 7f84ce400640 1 --2- 172.21.15.2:0/4001295902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104e50 0x7f84c810b6e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f84c0009ca0 tx=0x7f84c002c100 comp rx=0 tx=0).stop 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.062+0000 7f84ce400640 1 -- 172.21.15.2:0/4001295902 shutdown_connections 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 --2- 172.21.15.2:0/4001295902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104e50 0x7f84c810b6e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 --2- 172.21.15.2:0/4001295902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c8104490 0x7f84c8104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 --2- 172.21.15.2:0/4001295902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c81032a0 0x7f84c81036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 -- 172.21.15.2:0/4001295902 >> 172.21.15.2:0/4001295902 conn(0x7f84c80fe830 msgr2=0x7f84c8100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 -- 172.21.15.2:0/4001295902 shutdown_connections 2024-03-11T18:33:16.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.063+0000 7f84ce400640 1 -- 172.21.15.2:0/4001295902 wait complete. 2024-03-11T18:33:16.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.064+0000 7f84ce400640 1 Processor -- start 2024-03-11T18:33:16.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 -- start start 2024-03-11T18:33:16.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c81032a0 0x7f84c8199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104490 0x7f84c81997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81a0d70 con 0x7f84c81032a0 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81a0ee0 con 0x7f84c8104490 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84ce400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81a10a0 con 0x7f84c8104e50 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84cda00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84cda00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48216/0 (socket says 172.21.15.2:48216) 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.065+0000 7f84cda00640 1 -- 172.21.15.2:0/2690577957 learned_addr learned my addr 172.21.15.2:0/2690577957 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cd000640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c81032a0 0x7f84c8199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84c7e00640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104490 0x7f84c81997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 -- 172.21.15.2:0/2690577957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104490 msgr2=0x7f84c81997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104490 0x7f84c81997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 -- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c81032a0 msgr2=0x7f84c8199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c81032a0 0x7f84c8199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 -- 172.21.15.2:0/2690577957 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c0009900 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.066+0000 7f84cda00640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84c002c640 tx=0x7f84c0032fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.067+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0034640 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.067+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c81a1320 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.067+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84c0003e50 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.067+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0004840 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.067+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c81a1820 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.068+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c810a4f0 con 0x7f84c8104e50 2024-03-11T18:33:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.070+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f84c000a040 con 0x7f84c8104e50 2024-03-11T18:33:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.072+0000 7f84c5600640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 0x7f84b00a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.072+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f84c00f00b0 con 0x7f84c8104e50 2024-03-11T18:33:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.072+0000 7f84cd000640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 0x7f84b00a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.073+0000 7f84cd000640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 0x7f84b00a3f60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f84c81042f0 tx=0x7f84b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:16.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.076+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f84c00b8f00 con 0x7f84c8104e50 2024-03-11T18:33:16.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.280+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c802cda0 con 0x7f84c8104e50 2024-03-11T18:33:16.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.282+0000 7f84c5600640 1 -- 172.21.15.2:0/2690577957 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7f84c00bc5c0 con 0x7f84c8104e50 2024-03-11T18:33:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:16.284 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:33:16.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 msgr2=0x7f84b00a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 0x7f84b00a3f60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f84c81042f0 tx=0x7f84b4005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 msgr2=0x7f84c81a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84c002c640 tx=0x7f84c0032fc0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 shutdown_connections 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f84b00a1aa0 0x7f84b00a3f60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c8104e50 0x7f84c81a0830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c8104490 0x7f84c81997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 --2- 172.21.15.2:0/2690577957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c81032a0 0x7f84c8199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 >> 172.21.15.2:0/2690577957 conn(0x7f84c80fe830 msgr2=0x7f84c81076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 shutdown_connections 2024-03-11T18:33:16.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.285+0000 7f84ce400640 1 -- 172.21.15.2:0/2690577957 wait complete. 2024-03-11T18:33:16.306 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:33:16.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:33:16.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 -- 172.21.15.2:0/3409117597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740075b40 msgr2=0x7f1740075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 --2- 172.21.15.2:0/3409117597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740075b40 0x7f1740075f60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1730009900 tx=0x7f173002bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:16.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 -- 172.21.15.2:0/3409117597 shutdown_connections 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 --2- 172.21.15.2:0/3409117597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740077720 0x7f1740077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 --2- 172.21.15.2:0/3409117597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740076d80 0x7f17400771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 --2- 172.21.15.2:0/3409117597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740075b40 0x7f1740075f60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 -- 172.21.15.2:0/3409117597 >> 172.21.15.2:0/3409117597 conn(0x7f17400fe750 msgr2=0x7f1740100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 -- 172.21.15.2:0/3409117597 shutdown_connections 2024-03-11T18:33:16.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.528+0000 7f173fe00640 1 -- 172.21.15.2:0/3409117597 wait complete. 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.529+0000 7f173fe00640 1 Processor -- start 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 -- start start 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 0x7f174019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740076d80 0x7f174019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401a5160 con 0x7f1740077720 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401a52d0 con 0x7f1740075b40 2024-03-11T18:33:16.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17401a5490 con 0x7f1740076d80 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49936/0 (socket says 172.21.15.2:49936) 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173f400640 1 -- 172.21.15.2:0/4231152452 learned_addr learned my addr 172.21.15.2:0/4231152452 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 0x7f174019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.530+0000 7f173e000640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740076d80 0x7f174019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 -- 172.21.15.2:0/4231152452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740076d80 msgr2=0x7f174019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740076d80 0x7f174019dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 -- 172.21.15.2:0/4231152452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 msgr2=0x7f174019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 0x7f174019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 -- 172.21.15.2:0/4231152452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009560 con 0x7f1740077720 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173ea00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 0x7f174019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173f400640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f173800eb10 tx=0x7f173800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:16.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738004b20 con 0x7f1740077720 2024-03-11T18:33:16.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17401a5770 con 0x7f1740077720 2024-03-11T18:33:16.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1738004c80 con 0x7f1740077720 2024-03-11T18:33:16.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.531+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738010670 con 0x7f1740077720 2024-03-11T18:33:16.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.532+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17401a5cd0 con 0x7f1740077720 2024-03-11T18:33:16.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.533+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174010fb30 con 0x7f1740077720 2024-03-11T18:33:16.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.535+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f1738005020 con 0x7f1740077720 2024-03-11T18:33:16.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.536+0000 7f1737400640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 0x7f17240a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:16.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.536+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f1738014070 con 0x7f1740077720 2024-03-11T18:33:16.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.536+0000 7f173ea00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 0x7f17240a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:16.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.537+0000 7f173ea00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 0x7f17240a3f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1730009230 tx=0x7f1730005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:16.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.541+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1738096b90 con 0x7f1740077720 2024-03-11T18:33:16.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:16 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2690577957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:16.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:16 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2690577957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:16.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.744+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f174002ce40 con 0x7f1740077720 2024-03-11T18:33:16.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.745+0000 7f1737400640 1 -- 172.21.15.2:0/4231152452 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f173809a250 con 0x7f1740077720 2024-03-11T18:33:16.746 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:33:16.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 msgr2=0x7f17240a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 0x7f17240a3f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1730009230 tx=0x7f1730005fd0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 msgr2=0x7f17401a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f173800eb10 tx=0x7f173800c2d0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 shutdown_connections 2024-03-11T18:33:16.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f17240a1ab0 0x7f17240a3f70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1740077720 0x7f17401a4c20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1740076d80 0x7f174019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 --2- 172.21.15.2:0/4231152452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1740075b40 0x7f174019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.748+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 >> 172.21.15.2:0/4231152452 conn(0x7f17400fe750 msgr2=0x7f1740100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.749+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 shutdown_connections 2024-03-11T18:33:16.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.749+0000 7f173fe00640 1 -- 172.21.15.2:0/4231152452 wait complete. 2024-03-11T18:33:16.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:16 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2690577957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:16.771 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:33:16.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.996+0000 7fa1eb200640 1 -- 172.21.15.2:0/1785527863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec075b40 msgr2=0x7fa1ec075f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.996+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1785527863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec075b40 0x7fa1ec075f60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009900 tx=0x7fa1dc02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 -- 172.21.15.2:0/1785527863 shutdown_connections 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1785527863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec077720 0x7fa1ec077fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1785527863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec076d80 0x7fa1ec0771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1785527863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec075b40 0x7fa1ec075f60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 -- 172.21.15.2:0/1785527863 >> 172.21.15.2:0/1785527863 conn(0x7fa1ec0fe750 msgr2=0x7fa1ec100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 -- 172.21.15.2:0/1785527863 shutdown_connections 2024-03-11T18:33:16.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.997+0000 7fa1eb200640 1 -- 172.21.15.2:0/1785527863 wait complete. 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 Processor -- start 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 -- start start 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec076d80 0x7fa1ec19dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec077720 0x7fa1ec1a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1a51a0 con 0x7fa1ec076d80 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1a5310 con 0x7fa1ec077720 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.998+0000 7fa1eb200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1a54d0 con 0x7fa1ec075b40 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec076d80 0x7fa1ec19dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48258/0 (socket says 172.21.15.2:48258) 2024-03-11T18:33:17.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 -- 172.21.15.2:0/1405036123 learned_addr learned my addr 172.21.15.2:0/1405036123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1ea800640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec077720 0x7fa1ec1a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 -- 172.21.15.2:0/1405036123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec077720 msgr2=0x7fa1ec1a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec077720 0x7fa1ec1a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 -- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec076d80 msgr2=0x7fa1ec19dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec076d80 0x7fa1ec19dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 -- 172.21.15.2:0/1405036123 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc009560 con 0x7fa1ec075b40 2024-03-11T18:33:17.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:16.999+0000 7fa1e9e00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc02c840 tx=0x7fa1dc0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.000+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0044c0 con 0x7fa1ec075b40 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.000+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc0052d0 con 0x7fa1ec075b40 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.000+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec1a5750 con 0x7fa1ec075b40 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.000+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc003e80 con 0x7fa1ec075b40 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.000+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec1a5cb0 con 0x7fa1ec075b40 2024-03-11T18:33:17.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.001+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec04eee0 con 0x7fa1ec075b40 2024-03-11T18:33:17.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.003+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa1dc005440 con 0x7fa1ec075b40 2024-03-11T18:33:17.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.004+0000 7fa1e2a00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 0x7fa1d00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.004+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fa1dc0f0110 con 0x7fa1ec075b40 2024-03-11T18:33:17.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.005+0000 7fa1e3e00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 0x7fa1d00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.005+0000 7fa1e3e00640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 0x7fa1d00a3f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec19ebc0 tx=0x7fa1d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.009+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1dc0b9020 con 0x7fa1ec075b40 2024-03-11T18:33:17.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.213+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1ec02cdd0 con 0x7fa1ec075b40 2024-03-11T18:33:17.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.214+0000 7fa1e2a00640 1 -- 172.21.15.2:0/1405036123 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1dc043020 con 0x7fa1ec075b40 2024-03-11T18:33:17.215 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:33:17.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 msgr2=0x7fa1d00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 0x7fa1d00a3f70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec19ebc0 tx=0x7fa1d4005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:17.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 msgr2=0x7fa1ec19d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc02c840 tx=0x7fa1dc0040c0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 shutdown_connections 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa1d00a1ab0 0x7fa1d00a3f70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1ec077720 0x7fa1ec1a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1ec076d80 0x7fa1ec19dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 --2- 172.21.15.2:0/1405036123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1ec075b40 0x7fa1ec19d700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 >> 172.21.15.2:0/1405036123 conn(0x7fa1ec0fe750 msgr2=0x7fa1ec100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 shutdown_connections 2024-03-11T18:33:17.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.217+0000 7fa1eb200640 1 -- 172.21.15.2:0/1405036123 wait complete. 2024-03-11T18:33:17.240 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:33:17.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.466+0000 7f7563e00640 1 -- 172.21.15.2:0/1960260052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 msgr2=0x7f7564104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.466+0000 7f7563e00640 1 --2- 172.21.15.2:0/1960260052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564104850 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f754c009900 tx=0x7f754c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 -- 172.21.15.2:0/1960260052 shutdown_connections 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 --2- 172.21.15.2:0/1960260052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7564104d90 0x7f756410b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 --2- 172.21.15.2:0/1960260052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564104850 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 --2- 172.21.15.2:0/1960260052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75641031e0 0x7f7564103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 -- 172.21.15.2:0/1960260052 >> 172.21.15.2:0/1960260052 conn(0x7f75640fe750 msgr2=0x7f7564100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 -- 172.21.15.2:0/1960260052 shutdown_connections 2024-03-11T18:33:17.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.467+0000 7f7563e00640 1 -- 172.21.15.2:0/1960260052 wait complete. 2024-03-11T18:33:17.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.468+0000 7f7563e00640 1 Processor -- start 2024-03-11T18:33:17.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.468+0000 7f7563e00640 1 -- start start 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 0x7f7564199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7564104d90 0x7f75641a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75641a0d40 con 0x7f75641031e0 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75641a0eb0 con 0x7f75641043d0 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75641a1070 con 0x7f7564104d90 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7562a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 0x7f7564199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7562000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7563400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7564104d90 0x7f75641a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7562000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45942/0 (socket says 172.21.15.2:45942) 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7562000640 1 -- 172.21.15.2:0/4103189764 learned_addr learned my addr 172.21.15.2:0/4103189764 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.469+0000 7f7562a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 0x7f7564199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42864/0 (socket says 172.21.15.2:42864) 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 -- 172.21.15.2:0/4103189764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7564104d90 msgr2=0x7f75641a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7564104d90 0x7f75641a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 -- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 msgr2=0x7f7564199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 0x7f7564199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 -- 172.21.15.2:0/4103189764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c009560 con 0x7f75641043d0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7562000640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c0043d0 con 0x7f75641043d0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f754c0052d0 con 0x7f75641043d0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75641a12f0 con 0x7f75641043d0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c003cc0 con 0x7f75641043d0 2024-03-11T18:33:17.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.470+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75641a1850 con 0x7f75641043d0 2024-03-11T18:33:17.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.471+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756410a520 con 0x7f75641043d0 2024-03-11T18:33:17.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.473+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f754c005440 con 0x7f75641043d0 2024-03-11T18:33:17.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.475+0000 7f755b400640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 0x7f75480a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.475+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f754c0efc70 con 0x7f75641043d0 2024-03-11T18:33:17.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.475+0000 7f7562a00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 0x7f75480a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.476+0000 7f7562a00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 0x7f75480a3f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7564104230 tx=0x7f7554005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.477+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f754c0b8a60 con 0x7f75641043d0 2024-03-11T18:33:17.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:17 smithi002 ceph-mon[31328]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:17.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4231152452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:17 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1405036123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:17 smithi023 ceph-mon[36406]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:17.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4231152452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:17 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1405036123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:17 smithi165 ceph-mon[36634]: pgmap v409: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:17.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4231152452' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:17 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1405036123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:17.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.678+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f756410ee40 con 0x7f75641043d0 2024-03-11T18:33:17.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.679+0000 7f755b400640 1 -- 172.21.15.2:0/4103189764 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f754c0bc120 con 0x7f75641043d0 2024-03-11T18:33:17.680 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:33:17.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.681+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 msgr2=0x7f75480a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.681+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 0x7f75480a3f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7564104230 tx=0x7f7554005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:17.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 msgr2=0x7f7564199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f754c006010 tx=0x7f754c005e80 comp rx=0 tx=0).stop 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 shutdown_connections 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f75480a1ab0 0x7f75480a3f70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7564104d90 0x7f75641a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75641043d0 0x7f7564199790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 --2- 172.21.15.2:0/4103189764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75641031e0 0x7f7564199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 >> 172.21.15.2:0/4103189764 conn(0x7f75640fe750 msgr2=0x7f75641076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 shutdown_connections 2024-03-11T18:33:17.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.682+0000 7f7563e00640 1 -- 172.21.15.2:0/4103189764 wait complete. 2024-03-11T18:33:17.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 -- 172.21.15.2:0/2353100524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 msgr2=0x7f1d941037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 --2- 172.21.15.2:0/2353100524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d941037d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009ca0 tx=0x7f1d8402c0f0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 -- 172.21.15.2:0/2353100524 shutdown_connections 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 --2- 172.21.15.2:0/2353100524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d9410b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 --2- 172.21.15.2:0/2353100524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 0x7f1d94104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 --2- 172.21.15.2:0/2353100524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d941037d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 -- 172.21.15.2:0/2353100524 >> 172.21.15.2:0/2353100524 conn(0x7f1d940fe940 msgr2=0x7f1d94100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 -- 172.21.15.2:0/2353100524 shutdown_connections 2024-03-11T18:33:17.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.931+0000 7f1d99a00640 1 -- 172.21.15.2:0/2353100524 wait complete. 2024-03-11T18:33:17.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 Processor -- start 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 -- start start 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d94199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 0x7f1d94199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d941a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.932+0000 7f1d99a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d941a0f40 con 0x7f1d941033b0 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d99a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d941a10b0 con 0x7f1d94104f60 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d99a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d941a1270 con 0x7f1d941045a0 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d94199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d94199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42880/0 (socket says 172.21.15.2:42880) 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d92a00640 1 -- 172.21.15.2:0/4136710827 learned_addr learned my addr 172.21.15.2:0/4136710827 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d941a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d92000640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 0x7f1d94199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 -- 172.21.15.2:0/4136710827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 msgr2=0x7f1d94199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 0x7f1d94199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 -- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 msgr2=0x7f1d94199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d94199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.933+0000 7f1d93400640 1 -- 172.21.15.2:0/4136710827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c009490 con 0x7f1d94104f60 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d93400640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d941a0a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00e970 tx=0x7f1d8c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c00ccc0 con 0x7f1d94104f60 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d8c00ce20 con 0x7f1d94104f60 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c010430 con 0x7f1d94104f60 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c009490 con 0x7f1d94104f60 2024-03-11T18:33:17.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.934+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d84009900 con 0x7f1d94104f60 2024-03-11T18:33:17.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.935+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9410a7a0 con 0x7f1d94104f60 2024-03-11T18:33:17.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.937+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f1d8c004020 con 0x7f1d94104f60 2024-03-11T18:33:17.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.938+0000 7f1d8b400640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 0x7f1d780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:17.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.939+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f1d8c014070 con 0x7f1d94104f60 2024-03-11T18:33:17.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.939+0000 7f1d92a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 0x7f1d780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:17.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.940+0000 7f1d92a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 0x7f1d780a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d84005e30 tx=0x7f1d84004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:17.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:17.941+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1d8c073d20 con 0x7f1d94104f60 2024-03-11T18:33:18.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.142+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1d9410edc0 con 0x7f1d94104f60 2024-03-11T18:33:18.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.143+0000 7f1d8b400640 1 -- 172.21.15.2:0/4136710827 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1d8c096b10 con 0x7f1d94104f60 2024-03-11T18:33:18.145 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.145+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 msgr2=0x7f1d780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.145+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 0x7f1d780a3ef0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d84005e30 tx=0x7f1d84004a40 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 msgr2=0x7f1d941a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d941a0a00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00e970 tx=0x7f1d8c00ee30 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 shutdown_connections 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1d780a1a30 0x7f1d780a3ef0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d94104f60 0x7f1d941a0a00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1d941045a0 0x7f1d94199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 --2- 172.21.15.2:0/4136710827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d941033b0 0x7f1d94199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 >> 172.21.15.2:0/4136710827 conn(0x7f1d940fe940 msgr2=0x7f1d94107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:18.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 shutdown_connections 2024-03-11T18:33:18.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.146+0000 7f1d99a00640 1 -- 172.21.15.2:0/4136710827 wait complete. 2024-03-11T18:33:18.168 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:33:18.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 -- 172.21.15.2:0/649266913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13801023b0 msgr2=0x7f1380102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 --2- 172.21.15.2:0/649266913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13801023b0 0x7f1380102830 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1368009900 tx=0x7f136802bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:18.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 -- 172.21.15.2:0/649266913 shutdown_connections 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 --2- 172.21.15.2:0/649266913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1380102d70 0x7f1380109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 --2- 172.21.15.2:0/649266913 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13801023b0 0x7f1380102830 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 --2- 172.21.15.2:0/649266913 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801011c0 0x7f13801015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 -- 172.21.15.2:0/649266913 >> 172.21.15.2:0/649266913 conn(0x7f13800fc710 msgr2=0x7f13800feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.395+0000 7f1387600640 1 -- 172.21.15.2:0/649266913 shutdown_connections 2024-03-11T18:33:18.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.396+0000 7f1387600640 1 -- 172.21.15.2:0/649266913 wait complete. 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 Processor -- start 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 -- start start 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 0x7f1380074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1380102d70 0x7f1380073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1380074210 con 0x7f1380102d70 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1380074380 con 0x7f13801011c0 2024-03-11T18:33:18.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.397+0000 7f1387600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13801a1270 con 0x7f13801023b0 2024-03-11T18:33:18.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1385600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1380102d70 0x7f1380073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59450/0 (socket says 172.21.15.2:59450) 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 -- 172.21.15.2:0/450784728 learned_addr learned my addr 172.21.15.2:0/450784728 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1384c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 0x7f1380074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 -- 172.21.15.2:0/450784728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 msgr2=0x7f1380074f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 0x7f1380074f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 -- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1380102d70 msgr2=0x7f1380073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1380102d70 0x7f1380073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 -- 172.21.15.2:0/450784728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368009560 con 0x7f13801023b0 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1384c00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 0x7f1380074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.398+0000 7f1377e00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1368006010 tx=0x7f1368005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:18.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.399+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680043d0 con 0x7f13801023b0 2024-03-11T18:33:18.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.399+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13801a14c0 con 0x7f13801023b0 2024-03-11T18:33:18.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.399+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13801a1a20 con 0x7f13801023b0 2024-03-11T18:33:18.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.399+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13680052d0 con 0x7f13801023b0 2024-03-11T18:33:18.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.399+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368003c90 con 0x7f13801023b0 2024-03-11T18:33:18.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.400+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c005290 con 0x7f13801023b0 2024-03-11T18:33:18.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.402+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f1368005440 con 0x7f13801023b0 2024-03-11T18:33:18.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.403+0000 7f137e000640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 0x7f13600a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.403+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f13680ef6d0 con 0x7f13801023b0 2024-03-11T18:33:18.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.403+0000 7f1384c00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 0x7f13600a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.404+0000 7f1384c00640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 0x7f13600a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1380102210 tx=0x7f1370005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:18.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.407+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f13680b84c0 con 0x7f13801023b0 2024-03-11T18:33:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4103189764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:18 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/4136710827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.650+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135c005ac0 con 0x7f13801023b0 2024-03-11T18:33:18.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.651+0000 7f137e000640 1 -- 172.21.15.2:0/450784728 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f13680bbb80 con 0x7f13801023b0 2024-03-11T18:33:18.653 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:18.653 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:33:18.653 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:33:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:18 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4103189764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:18 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/4136710827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:18 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4103189764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:18 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/4136710827' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 msgr2=0x7f13600a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 0x7f13600a3ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1380102210 tx=0x7f1370005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 msgr2=0x7f1380073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1368006010 tx=0x7f1368005e80 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 shutdown_connections 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f13600a1a30 0x7f13600a3ef0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1380102d70 0x7f1380073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13801023b0 0x7f1380073570 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 --2- 172.21.15.2:0/450784728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13801011c0 0x7f1380074f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 >> 172.21.15.2:0/450784728 conn(0x7f13800fc710 msgr2=0x7f13801056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 shutdown_connections 2024-03-11T18:33:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.654+0000 7f1387600640 1 -- 172.21.15.2:0/450784728 wait complete. 2024-03-11T18:33:18.677 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:33:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 -- 172.21.15.2:0/3055367690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 msgr2=0x7ff000107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 --2- 172.21.15.2:0/3055367690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff000107be0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7feff0009900 tx=0x7feff002bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 -- 172.21.15.2:0/3055367690 shutdown_connections 2024-03-11T18:33:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 --2- 172.21.15.2:0/3055367690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 0x7ff00010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 --2- 172.21.15.2:0/3055367690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 0x7ff000108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 --2- 172.21.15.2:0/3055367690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff000107be0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 -- 172.21.15.2:0/3055367690 >> 172.21.15.2:0/3055367690 conn(0x7ff000076780 msgr2=0x7ff000078ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 -- 172.21.15.2:0/3055367690 shutdown_connections 2024-03-11T18:33:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.902+0000 7ff005400640 1 -- 172.21.15.2:0/3055367690 wait complete. 2024-03-11T18:33:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.903+0000 7ff005400640 1 Processor -- start 2024-03-11T18:33:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.903+0000 7ff005400640 1 -- start start 2024-03-11T18:33:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 0x7ff00019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 0x7ff0001a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a53a0 con 0x7ff0001077c0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a5510 con 0x7ff000109370 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7ff005400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001a56d0 con 0x7ff0001089b0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7feffe800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7feffe800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42912/0 (socket says 172.21.15.2:42912) 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7feffe800640 1 -- 172.21.15.2:0/1727270733 learned_addr learned my addr 172.21.15.2:0/1727270733 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7feffde00640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 0x7ff00019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.904+0000 7fefff200640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 0x7ff0001a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 -- 172.21.15.2:0/1727270733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 msgr2=0x7ff00019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 0x7ff00019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 -- 172.21.15.2:0/1727270733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 msgr2=0x7ff0001a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 0x7ff0001a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 -- 172.21.15.2:0/1727270733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0009560 con 0x7ff0001077c0 2024-03-11T18:33:18.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feffe800640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feff0005ab0 tx=0x7feff0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0004aa0 con 0x7ff0001077c0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feff00052d0 con 0x7ff0001077c0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0003e80 con 0x7ff0001077c0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.905+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0001a5950 con 0x7ff0001077c0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.906+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0001a5e60 con 0x7ff0001077c0 2024-03-11T18:33:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.907+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00010eb60 con 0x7ff0001077c0 2024-03-11T18:33:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.909+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7feff0005440 con 0x7ff0001077c0 2024-03-11T18:33:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.911+0000 7feff7400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 0x7fefe40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.911+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7feff00efa40 con 0x7ff0001077c0 2024-03-11T18:33:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.911+0000 7feffde00640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 0x7fefe40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.912+0000 7feffde00640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 0x7fefe40a3ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff00019edc0 tx=0x7fefe8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:18.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:18.917+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feff00b8830 con 0x7ff0001077c0 2024-03-11T18:33:19.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.120+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff00002cdd0 con 0x7ff0001077c0 2024-03-11T18:33:19.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.124+0000 7feff7400640 1 -- 172.21.15.2:0/1727270733 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7feff00bbef0 con 0x7ff0001077c0 2024-03-11T18:33:19.126 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:19.127 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:33:19.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 msgr2=0x7fefe40a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 0x7fefe40a3ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff00019edc0 tx=0x7fefe8009500 comp rx=0 tx=0).stop 2024-03-11T18:33:19.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 msgr2=0x7ff00019d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feff0005ab0 tx=0x7feff0004270 comp rx=0 tx=0).stop 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 shutdown_connections 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fefe40a1a30 0x7fefe40a3ef0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff000109370 0x7ff0001a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0001089b0 0x7ff00019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 --2- 172.21.15.2:0/1727270733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0001077c0 0x7ff00019d8b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 >> 172.21.15.2:0/1727270733 conn(0x7ff000076780 msgr2=0x7ff00010bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 shutdown_connections 2024-03-11T18:33:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.127+0000 7ff005400640 1 -- 172.21.15.2:0/1727270733 wait complete. 2024-03-11T18:33:19.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:33:19.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:33:19.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.384+0000 7f8318800640 1 -- 172.21.15.2:0/834941307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 msgr2=0x7f8310077fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.384+0000 7f8318800640 1 --2- 172.21.15.2:0/834941307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f8310077fc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f830c009900 tx=0x7f830c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 -- 172.21.15.2:0/834941307 shutdown_connections 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 --2- 172.21.15.2:0/834941307 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f8310077fc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 --2- 172.21.15.2:0/834941307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 0x7f83100771e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 --2- 172.21.15.2:0/834941307 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f8310075f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 -- 172.21.15.2:0/834941307 >> 172.21.15.2:0/834941307 conn(0x7f83100fe940 msgr2=0x7f8310100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 -- 172.21.15.2:0/834941307 shutdown_connections 2024-03-11T18:33:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.385+0000 7f8318800640 1 -- 172.21.15.2:0/834941307 wait complete. 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 Processor -- start 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 -- start start 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 0x7f831019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f83101a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83101a5300 con 0x7f8310075b40 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83101a5470 con 0x7f8310077720 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.386+0000 7f8318800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83101a5630 con 0x7f8310076d80 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42932/0 (socket says 172.21.15.2:42932) 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 -- 172.21.15.2:0/1050611828 learned_addr learned my addr 172.21.15.2:0/1050611828 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315400640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 0x7f831019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8316800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f83101a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 -- 172.21.15.2:0/1050611828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 msgr2=0x7f831019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 0x7f831019dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 -- 172.21.15.2:0/1050611828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 msgr2=0x7f83101a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f83101a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.387+0000 7f8315e00640 1 -- 172.21.15.2:0/1050611828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c009560 con 0x7f8310075b40 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8315e00640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f830800ebc0 tx=0x7f830800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83080099b0 con 0x7f8310075b40 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f830800eec0 con 0x7f8310075b40 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83101a5910 con 0x7f8310075b40 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308018b60 con 0x7f8310075b40 2024-03-11T18:33:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.388+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83101a5e20 con 0x7f8310075b40 2024-03-11T18:33:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.389+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831010fe60 con 0x7f8310075b40 2024-03-11T18:33:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.391+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f8308018cc0 con 0x7f8310075b40 2024-03-11T18:33:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.393+0000 7f8306a00640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 0x7f82fc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.393+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f8308014070 con 0x7f8310075b40 2024-03-11T18:33:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.393+0000 7f8315400640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 0x7f82fc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.394+0000 7f8315400640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 0x7f82fc0a3ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f831019ed20 tx=0x7f8300005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.395+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83080971e0 con 0x7f8310075b40 2024-03-11T18:33:19.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:19 smithi002 ceph-mon[31328]: pgmap v410: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:19.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/450784728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:19 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1727270733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.597+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f831002cee0 con 0x7f8310075b40 2024-03-11T18:33:19.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.598+0000 7f8306a00640 1 -- 172.21.15.2:0/1050611828 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8308026020 con 0x7f8310075b40 2024-03-11T18:33:19.599 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:33:19.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 msgr2=0x7f82fc0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 0x7f82fc0a3ef0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f831019ed20 tx=0x7f8300005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 msgr2=0x7f831019d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f830800ebc0 tx=0x7f830800c6a0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 shutdown_connections 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f82fc0a1a30 0x7f82fc0a3ef0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8310077720 0x7f83101a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8310076d80 0x7f831019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 --2- 172.21.15.2:0/1050611828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8310075b40 0x7f831019d810 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 >> 172.21.15.2:0/1050611828 conn(0x7f83100fe940 msgr2=0x7f83100fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 shutdown_connections 2024-03-11T18:33:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.601+0000 7f8318800640 1 -- 172.21.15.2:0/1050611828 wait complete. 2024-03-11T18:33:19.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:33:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:19 smithi165 ceph-mon[36634]: pgmap v410: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:19 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/450784728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:19 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1727270733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:19 smithi023 ceph-mon[36406]: pgmap v410: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:19 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/450784728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:19 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1727270733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 -- 172.21.15.2:0/2426735685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced8100bc0 msgr2=0x7fced8101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 --2- 172.21.15.2:0/2426735685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced8100bc0 0x7fced8101040 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fced0009be0 tx=0x7fced002bf80 comp rx=0 tx=0).stop 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 -- 172.21.15.2:0/2426735685 shutdown_connections 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 --2- 172.21.15.2:0/2426735685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced8100bc0 0x7fced8101040 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 --2- 172.21.15.2:0/2426735685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 0x7fced8100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 --2- 172.21.15.2:0/2426735685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced80ff8c0 0x7fced80ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 -- 172.21.15.2:0/2426735685 >> 172.21.15.2:0/2426735685 conn(0x7fced80fb460 msgr2=0x7fced80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 -- 172.21.15.2:0/2426735685 shutdown_connections 2024-03-11T18:33:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.857+0000 7fcedf000640 1 -- 172.21.15.2:0/2426735685 wait complete. 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.858+0000 7fcedf000640 1 Processor -- start 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.858+0000 7fcedf000640 1 -- start start 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 0x7fced8199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 0x7fced81997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 0x7fced81a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced81a0d00 con 0x7fced8100220 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced81a0e70 con 0x7fced8100bc0 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedf000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced81a1030 con 0x7fced80ff8c0 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcedd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 0x7fced81997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fceddc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 0x7fced8199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 0x7fced81a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fceddc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 0x7fced8199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59502/0 (socket says 172.21.15.2:59502) 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fceddc00640 1 -- 172.21.15.2:0/3367544597 learned_addr learned my addr 172.21.15.2:0/3367544597 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 -- 172.21.15.2:0/3367544597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 msgr2=0x7fced8199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 0x7fced8199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 -- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 msgr2=0x7fced81997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 0x7fced81997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.859+0000 7fcede600640 1 -- 172.21.15.2:0/3367544597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4009560 con 0x7fced8100bc0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fcede600640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 0x7fced81a07c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fced0036040 tx=0x7fced0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced00045f0 con 0x7fced8100bc0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fced0004750 con 0x7fced8100bc0 2024-03-11T18:33:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0034730 con 0x7fced8100bc0 2024-03-11T18:33:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced0009840 con 0x7fced8100bc0 2024-03-11T18:33:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.860+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced81a15e0 con 0x7fced8100bc0 2024-03-11T18:33:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.861+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced810a610 con 0x7fced8100bc0 2024-03-11T18:33:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.863+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fced0033c50 con 0x7fced8100bc0 2024-03-11T18:33:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.865+0000 7fced6000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 0x7fcec00a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.865+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fced00f0700 con 0x7fced8100bc0 2024-03-11T18:33:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.865+0000 7fceddc00640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 0x7fcec00a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.866+0000 7fceddc00640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 0x7fcec00a3f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009c90 tx=0x7fcecc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:19.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:19.867+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fced00b94f0 con 0x7fced8100bc0 2024-03-11T18:33:20.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.070+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fced810ec30 con 0x7fced8100bc0 2024-03-11T18:33:20.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.071+0000 7fced6000640 1 -- 172.21.15.2:0/3367544597 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fced0043020 con 0x7fced8100bc0 2024-03-11T18:33:20.072 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 msgr2=0x7fcec00a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 0x7fcec00a3f70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009c90 tx=0x7fcecc005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 msgr2=0x7fced81a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 0x7fced81a07c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fced0036040 tx=0x7fced0004270 comp rx=0 tx=0).stop 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 shutdown_connections 2024-03-11T18:33:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcec00a1ab0 0x7fcec00a3f70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fced8100bc0 0x7fced81a07c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced8100220 0x7fced81997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 --2- 172.21.15.2:0/3367544597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fced80ff8c0 0x7fced8199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.073+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 >> 172.21.15.2:0/3367544597 conn(0x7fced80fb460 msgr2=0x7fced80fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.074+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 shutdown_connections 2024-03-11T18:33:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.074+0000 7fcedf000640 1 -- 172.21.15.2:0/3367544597 wait complete. 2024-03-11T18:33:20.094 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:33:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 -- 172.21.15.2:0/1368436021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb0073be0 msgr2=0x7fecb0074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 --2- 172.21.15.2:0/1368436021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb0073be0 0x7fecb0074060 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009930 tx=0x7fec9c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 -- 172.21.15.2:0/1368436021 shutdown_connections 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 --2- 172.21.15.2:0/1368436021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb010d760 0x7fecb010fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 --2- 172.21.15.2:0/1368436021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb0073be0 0x7fecb0074060 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 --2- 172.21.15.2:0/1368436021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb00751c0 0x7fecb0073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 -- 172.21.15.2:0/1368436021 >> 172.21.15.2:0/1368436021 conn(0x7fecb00fc700 msgr2=0x7fecb00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 -- 172.21.15.2:0/1368436021 shutdown_connections 2024-03-11T18:33:20.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.325+0000 7fecb8400640 1 -- 172.21.15.2:0/1368436021 wait complete. 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.326+0000 7fecb8400640 1 Processor -- start 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.326+0000 7fecb8400640 1 -- start start 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb0073be0 0x7fecb019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb010d760 0x7fecb01a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb01a5410 con 0x7fecb0073be0 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb01a5580 con 0x7fecb00751c0 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb8400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb01a5740 con 0x7fecb010d760 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb5000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb5000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46040/0 (socket says 172.21.15.2:46040) 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb5000640 1 -- 172.21.15.2:0/1197215909 learned_addr learned my addr 172.21.15.2:0/1197215909 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:20.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb5a00640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb0073be0 0x7fecb019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.327+0000 7fecb6400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb010d760 0x7fecb01a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 -- 172.21.15.2:0/1197215909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb010d760 msgr2=0x7fecb01a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb010d760 0x7fecb01a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 -- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb0073be0 msgr2=0x7fecb019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb0073be0 0x7fecb019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 -- 172.21.15.2:0/1197215909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c009560 con 0x7fecb00751c0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb5000640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000c00 tx=0x7fec9c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0044c0 con 0x7fecb00751c0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec9c0052d0 con 0x7fecb00751c0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c003e50 con 0x7fecb00751c0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.328+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb01a59c0 con 0x7fecb00751c0 2024-03-11T18:33:20.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.329+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb01a5e80 con 0x7fecb00751c0 2024-03-11T18:33:20.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.330+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb0103820 con 0x7fecb00751c0 2024-03-11T18:33:20.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.332+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fec9c005440 con 0x7fecb00751c0 2024-03-11T18:33:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.334+0000 7fecae000640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 0x7fec980a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.334+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fec9c0f03d0 con 0x7fecb00751c0 2024-03-11T18:33:20.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.334+0000 7fecb5a00640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 0x7fec980a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.335+0000 7fecb5a00640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 0x7fec980a3f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feca4009bd0 tx=0x7feca4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:20.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.336+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec9c0b91c0 con 0x7fecb00751c0 2024-03-11T18:33:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.545+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fecb0113420 con 0x7fecb00751c0 2024-03-11T18:33:20.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.546+0000 7fecae000640 1 -- 172.21.15.2:0/1197215909 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec9c0430a0 con 0x7fecb00751c0 2024-03-11T18:33:20.548 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 msgr2=0x7fec980a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 0x7fec980a3f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7feca4009bd0 tx=0x7feca4005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 msgr2=0x7fecb019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000c00 tx=0x7fec9c0040c0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 shutdown_connections 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec980a1ab0 0x7fec980a3f70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecb010d760 0x7fecb01a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecb00751c0 0x7fecb019de60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 --2- 172.21.15.2:0/1197215909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb0073be0 0x7fecb019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 >> 172.21.15.2:0/1197215909 conn(0x7fecb00fc700 msgr2=0x7fecb01009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 shutdown_connections 2024-03-11T18:33:20.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.549+0000 7fecb8400640 1 -- 172.21.15.2:0/1197215909 wait complete. 2024-03-11T18:33:20.569 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:33:20.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:20 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1050611828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.654 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:20 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3367544597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:20 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1050611828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:20 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3367544597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:20 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1050611828' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:20 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3367544597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 -- 172.21.15.2:0/2788002910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 msgr2=0x7f1c0c104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 --2- 172.21.15.2:0/2788002910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c104850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009900 tx=0x7f1bf402bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 -- 172.21.15.2:0/2788002910 shutdown_connections 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 --2- 172.21.15.2:0/2788002910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c10b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 --2- 172.21.15.2:0/2788002910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c104850 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 --2- 172.21.15.2:0/2788002910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 -- 172.21.15.2:0/2788002910 >> 172.21.15.2:0/2788002910 conn(0x7f1c0c0fe750 msgr2=0x7f1c0c100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 -- 172.21.15.2:0/2788002910 shutdown_connections 2024-03-11T18:33:20.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.800+0000 7f1c11000640 1 -- 172.21.15.2:0/2788002910 wait complete. 2024-03-11T18:33:20.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.801+0000 7f1c11000640 1 Processor -- start 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.801+0000 7f1c11000640 1 -- start start 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.801+0000 7f1c11000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c11000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c11000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c11000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1a0d80 con 0x7f1c0c104d90 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c11000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1a0ef0 con 0x7f1c0c1031e0 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c11000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1a10b0 con 0x7f1c0c1043d0 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42984/0 (socket says 172.21.15.2:42984) 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 -- 172.21.15.2:0/1770437614 learned_addr learned my addr 172.21.15.2:0/1770437614 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c03e00640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0b400640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 -- 172.21.15.2:0/1770437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 msgr2=0x7f1c0c1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 -- 172.21.15.2:0/1770437614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 msgr2=0x7f1c0c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0be00640 1 -- 172.21.15.2:0/1770437614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4009560 con 0x7f1c0c104d90 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.802+0000 7f1c0b400640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c0be00640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400eb10 tx=0x7f1c0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04004b20 con 0x7f1c0c104d90 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c1a1390 con 0x7f1c0c104d90 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c04004c80 con 0x7f1c0c104d90 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04010650 con 0x7f1c0c104d90 2024-03-11T18:33:20.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.803+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c1a1820 con 0x7f1c0c104d90 2024-03-11T18:33:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.804+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c10a4a0 con 0x7f1c0c104d90 2024-03-11T18:33:20.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.806+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f1c04005020 con 0x7f1c0c104d90 2024-03-11T18:33:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.807+0000 7f1c08c00640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 0x7f1bf00a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.808+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f1c04014070 con 0x7f1c0c104d90 2024-03-11T18:33:20.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.808+0000 7f1c0b400640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 0x7f1bf00a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:20.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.808+0000 7f1c0b400640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 0x7f1bf00a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c104230 tx=0x7f1bfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:20.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:20.812+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1c04096c80 con 0x7f1c0c104d90 2024-03-11T18:33:21.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.015+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1c0c02ce40 con 0x7f1c0c104d90 2024-03-11T18:33:21.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.016+0000 7f1c08c00640 1 -- 172.21.15.2:0/1770437614 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1c0409a340 con 0x7f1c0c104d90 2024-03-11T18:33:21.018 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:33:21.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 msgr2=0x7f1bf00a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 0x7f1bf00a3ef0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c104230 tx=0x7f1bfc005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 msgr2=0x7f1c0c1a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400eb10 tx=0x7f1c0400c2d0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 shutdown_connections 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f1bf00a1a30 0x7f1bf00a3ef0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c0c104d90 0x7f1c0c1a0840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c0c1043d0 0x7f1c0c1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 --2- 172.21.15.2:0/1770437614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c0c1031e0 0x7f1c0c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 >> 172.21.15.2:0/1770437614 conn(0x7f1c0c0fe750 msgr2=0x7f1c0c107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:21.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 shutdown_connections 2024-03-11T18:33:21.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.019+0000 7f1c11000640 1 -- 172.21.15.2:0/1770437614 wait complete. 2024-03-11T18:33:21.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:33:21.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.262+0000 7f51ebe00640 1 -- 172.21.15.2:0/715544952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec1032c0 msgr2=0x7f51ec1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.262+0000 7f51ebe00640 1 --2- 172.21.15.2:0/715544952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec1032c0 0x7f51ec1036e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009ca0 tx=0x7f51dc02c0f0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 -- 172.21.15.2:0/715544952 shutdown_connections 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 --2- 172.21.15.2:0/715544952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec104e70 0x7f51ec10b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 --2- 172.21.15.2:0/715544952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1044b0 0x7f51ec104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 --2- 172.21.15.2:0/715544952 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec1032c0 0x7f51ec1036e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 -- 172.21.15.2:0/715544952 >> 172.21.15.2:0/715544952 conn(0x7f51ec0fe850 msgr2=0x7f51ec100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 -- 172.21.15.2:0/715544952 shutdown_connections 2024-03-11T18:33:21.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.263+0000 7f51ebe00640 1 -- 172.21.15.2:0/715544952 wait complete. 2024-03-11T18:33:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 Processor -- start 2024-03-11T18:33:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 -- start start 2024-03-11T18:33:21.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 0x7f51ec1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 0x7f51ec1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec1a0e60 con 0x7f51ec1032c0 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec1a0fd0 con 0x7f51ec1044b0 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51ebe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec1a1190 con 0x7f51ec104e70 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.264+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42990/0 (socket says 172.21.15.2:42990) 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51eaa00640 1 -- 172.21.15.2:0/3840765903 learned_addr learned my addr 172.21.15.2:0/3840765903 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 0x7f51ec1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51eb400640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 0x7f51ec1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 -- 172.21.15.2:0/3840765903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 msgr2=0x7f51ec1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 0x7f51ec1a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 -- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 msgr2=0x7f51ec199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 -- 172.21.15.2:0/3840765903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc009900 con 0x7f51ec1044b0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51eaa00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51eb400640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 0x7f51ec1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.265+0000 7f51ea000640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 0x7f51ec1998b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e9f0 tx=0x7f51d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.266+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d400ccc0 con 0x7f51ec1044b0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.266+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec1a1470 con 0x7f51ec1044b0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.266+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51d400ce20 con 0x7f51ec1044b0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.266+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4018920 con 0x7f51ec1044b0 2024-03-11T18:33:21.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.266+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec1a19d0 con 0x7f51ec1044b0 2024-03-11T18:33:21.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.267+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec10a5e0 con 0x7f51ec1044b0 2024-03-11T18:33:21.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.269+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f51d4018a80 con 0x7f51ec1044b0 2024-03-11T18:33:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.270+0000 7f51f0a00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 0x7f51d00a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.270+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f51d4014070 con 0x7f51ec1044b0 2024-03-11T18:33:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.270+0000 7f51eaa00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 0x7f51d00a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.271+0000 7f51eaa00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 0x7f51d00a3f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51dc005e30 tx=0x7f51dc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:21.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.275+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51d4052d20 con 0x7f51ec1044b0 2024-03-11T18:33:21.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.515+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51ec10eca0 con 0x7f51ec1044b0 2024-03-11T18:33:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.516+0000 7f51f0a00640 1 -- 172.21.15.2:0/3840765903 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f51d40968f0 con 0x7f51ec1044b0 2024-03-11T18:33:21.518 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:21.519 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:31:01.455980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/3141279332","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":3141279332},{"type":"v1","addr":"172.21.15.165:6837","nonce":3141279332}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:33:21.519 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 24 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 msgr2=0x7f51d00a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 0x7f51d00a3f60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51dc005e30 tx=0x7f51dc004a40 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 msgr2=0x7f51ec1998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 0x7f51ec1998b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e9f0 tx=0x7f51d400eeb0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 shutdown_connections 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f51d00a1aa0 0x7f51d00a3f60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec104e70 0x7f51ec1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51ec1044b0 0x7f51ec1998b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 --2- 172.21.15.2:0/3840765903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51ec1032c0 0x7f51ec199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 >> 172.21.15.2:0/3840765903 conn(0x7f51ec0fe850 msgr2=0x7f51ec1077b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 shutdown_connections 2024-03-11T18:33:21.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.520+0000 7f51ebe00640 1 -- 172.21.15.2:0/3840765903 wait complete. 2024-03-11T18:33:21.543 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:33:21.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:21 smithi002 ceph-mon[31328]: pgmap v411: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:21.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:21 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1197215909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:21 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1770437614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:21 smithi165 ceph-mon[36634]: pgmap v411: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:21.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1197215909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:21 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1770437614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:21 smithi023 ceph-mon[36406]: pgmap v411: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-11T18:33:21.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:21 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1197215909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:21 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1770437614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:33:21.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 -- 172.21.15.2:0/183969151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104ff0 msgr2=0x7fcba410b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 --2- 172.21.15.2:0/183969151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104ff0 0x7fcba410b880 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009ca0 tx=0x7fcb9c02c100 comp rx=0 tx=0).stop 2024-03-11T18:33:21.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 -- 172.21.15.2:0/183969151 shutdown_connections 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 --2- 172.21.15.2:0/183969151 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104ff0 0x7fcba410b880 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 --2- 172.21.15.2:0/183969151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4104630 0x7fcba4104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 --2- 172.21.15.2:0/183969151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4103440 0x7fcba4103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 -- 172.21.15.2:0/183969151 >> 172.21.15.2:0/183969151 conn(0x7fcba40fe990 msgr2=0x7fcba4100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 -- 172.21.15.2:0/183969151 shutdown_connections 2024-03-11T18:33:21.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.765+0000 7fcbac200640 1 -- 172.21.15.2:0/183969151 wait complete. 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.766+0000 7fcbac200640 1 Processor -- start 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 -- start start 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4103440 0x7fcba4199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 0x7fcba41a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41a0f10 con 0x7fcba4104630 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41a1080 con 0x7fcba4104ff0 2024-03-11T18:33:21.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbac200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41a1240 con 0x7fcba4103440 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcba8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcbaa200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 0x7fcba41a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcba8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43006/0 (socket says 172.21.15.2:43006) 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcba8e00640 1 -- 172.21.15.2:0/1701735908 learned_addr learned my addr 172.21.15.2:0/1701735908 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.767+0000 7fcba9800640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4103440 0x7fcba4199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 -- 172.21.15.2:0/1701735908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4103440 msgr2=0x7fcba4199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4103440 0x7fcba4199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 -- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 msgr2=0x7fcba4199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 -- 172.21.15.2:0/1701735908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb94009490 con 0x7fcba4104ff0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcba8e00640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbaa200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 0x7fcba41a09d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c02c640 tx=0x7fcb9c009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c034850 con 0x7fcba4104ff0 2024-03-11T18:33:21.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb9c003e50 con 0x7fcba4104ff0 2024-03-11T18:33:21.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c040450 con 0x7fcba4104ff0 2024-03-11T18:33:21.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.768+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c009900 con 0x7fcba4104ff0 2024-03-11T18:33:21.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.769+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba41a1760 con 0x7fcba4104ff0 2024-03-11T18:33:21.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.770+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba410a700 con 0x7fcba4104ff0 2024-03-11T18:33:21.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.772+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fcb9c00a040 con 0x7fcba4104ff0 2024-03-11T18:33:21.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.773+0000 7fcba2000640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 0x7fcb880a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:21.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.773+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fcb9c0f0990 con 0x7fcba4104ff0 2024-03-11T18:33:21.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.773+0000 7fcba9800640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 0x7fcb880a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:21.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.774+0000 7fcba9800640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 0x7fcb880a3ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcba4104490 tx=0x7fcb94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:21.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.775+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb9c0b9930 con 0x7fcba4104ff0 2024-03-11T18:33:21.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.977+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcba410edb0 con 0x7fcba4104ff0 2024-03-11T18:33:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.979+0000 7fcba2000640 1 -- 172.21.15.2:0/1701735908 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+24731 (secure 0 0 0) 0x7fcb9c0bcff0 con 0x7fcba4104ff0 2024-03-11T18:33:21.981 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:33:21.981 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":95,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:32:29.901057+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:33:21.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.981+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 msgr2=0x7fcb880a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.981+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 0x7fcb880a3ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcba4104490 tx=0x7fcb94005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 msgr2=0x7fcba41a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 0x7fcba41a09d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c02c640 tx=0x7fcb9c009360 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 shutdown_connections 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fcb880a1a30 0x7fcb880a3ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcba4104ff0 0x7fcba41a09d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcba4104630 0x7fcba4199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 --2- 172.21.15.2:0/1701735908 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcba4103440 0x7fcba4199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 >> 172.21.15.2:0/1701735908 conn(0x7fcba40fe990 msgr2=0x7fcba41078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 shutdown_connections 2024-03-11T18:33:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:21.982+0000 7fcbac200640 1 -- 172.21.15.2:0/1701735908 wait complete. 2024-03-11T18:33:22.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-11T18:30:27.790111+0000', 'modified': '2024-03-11T18:31:01.455980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24550, 'mds_2': 24521}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6835/1605486605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6834', 'nonce': 1605486605}, {'type': 'v1', 'addr': '172.21.15.23:6835', 'nonce': 1605486605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6835/2758205355', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6834', 'nonce': 2758205355}, {'type': 'v1', 'addr': '172.21.15.165:6835', 'nonce': 2758205355}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.165:6837/3141279332', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6836', 'nonce': 3141279332}, {'type': 'v1', 'addr': '172.21.15.165:6837', 'nonce': 3141279332}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.23:6837/531049865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6836', 'nonce': 531049865}, {'type': 'v1', 'addr': '172.21.15.23:6837', 'nonce': 531049865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.165:6839/1329073865', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.165:6838', 'nonce': 1329073865}, {'type': 'v1', 'addr': '172.21.15.165:6839', 'nonce': 1329073865}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.23:6839/2250151518', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 2250151518}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 2250151518}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-11T18:33:22.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-11T18:33:22.006 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 -- 172.21.15.2:0/3912645870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401089b0 msgr2=0x7fec40108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 --2- 172.21.15.2:0/3912645870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401089b0 0x7fec40108e30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009860 tx=0x7fec2c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 -- 172.21.15.2:0/3912645870 shutdown_connections 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 --2- 172.21.15.2:0/3912645870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 0x7fec4010fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 --2- 172.21.15.2:0/3912645870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401089b0 0x7fec40108e30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 --2- 172.21.15.2:0/3912645870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401077c0 0x7fec40107be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 -- 172.21.15.2:0/3912645870 >> 172.21.15.2:0/3912645870 conn(0x7fec40076770 msgr2=0x7fec40078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 -- 172.21.15.2:0/3912645870 shutdown_connections 2024-03-11T18:33:22.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.237+0000 7fec48400640 1 -- 172.21.15.2:0/3912645870 wait complete. 2024-03-11T18:33:22.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.238+0000 7fec48400640 1 Processor -- start 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.238+0000 7fec48400640 1 -- start start 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401077c0 0x7fec4019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 0x7fec401a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec401a5390 con 0x7fec40109370 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec401a5500 con 0x7fec401089b0 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec48400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec401a56c0 con 0x7fec401077c0 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec45000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec46400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 0x7fec401a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec45000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46118/0 (socket says 172.21.15.2:46118) 2024-03-11T18:33:22.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec45000640 1 -- 172.21.15.2:0/1557366851 learned_addr learned my addr 172.21.15.2:0/1557366851 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.239+0000 7fec45a00640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401077c0 0x7fec4019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 -- 172.21.15.2:0/1557366851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401077c0 msgr2=0x7fec4019d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401077c0 0x7fec4019d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 -- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 msgr2=0x7fec401a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 0x7fec401a4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 -- 172.21.15.2:0/1557366851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38009560 con 0x7fec401089b0 2024-03-11T18:33:22.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec45000640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005940 tx=0x7fec2c035f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec3e000640 1 -- 172.21.15.2:0/1557366851 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c02c710 con 0x7fec401089b0 2024-03-11T18:33:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec3e000640 1 -- 172.21.15.2:0/1557366851 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec2c034910 con 0x7fec401089b0 2024-03-11T18:33:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c009490 con 0x7fec401089b0 2024-03-11T18:33:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.240+0000 7fec3e000640 1 -- 172.21.15.2:0/1557366851 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c032860 con 0x7fec401089b0 2024-03-11T18:33:22.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.241+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec401a5b20 con 0x7fec401089b0 2024-03-11T18:33:22.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 Processor -- start 2024-03-11T18:33:22.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 -- start start 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x55659292d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592894b30 con 0x55659292b0e0 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592892060 con 0x556592925f70 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec4b188740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55659288d840 con 0x556592925580 2024-03-11T18:33:22.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec3e000640 1 -- 172.21.15.2:0/1557366851 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fec2c03b040 con 0x7fec401089b0 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec46400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec46400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59594/0 (socket says 172.21.15.2:59594) 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.244+0000 7fec46400640 1 -- 172.21.15.2:0/3618574065 learned_addr learned my addr 172.21.15.2:0/3618574065 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec45000640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x55659292d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec45a00640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 -- 172.21.15.2:0/3618574065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 msgr2=0x55659292aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 -- 172.21.15.2:0/3618574065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 msgr2=0x55659292d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x55659292d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 -- 172.21.15.2:0/3618574065 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565928554c0 con 0x556592925580 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec45000640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x55659292d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec46400640 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fec38009d70 tx=0x7fec38011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a48e27de5361c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.245+0000 7fec36a00640 1 -- 172.21.15.2:0/3618574065 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38004230 con 0x556592925580 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec36a00640 1 -- 172.21.15.2:0/3618574065 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec38004390 con 0x556592925580 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec36a00640 1 -- 172.21.15.2:0/3618574065 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec38019a00 con 0x556592925580 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec3e000640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 0x7fec240a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 -- 172.21.15.2:0/3618574065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 msgr2=0x5565929259a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fec38009d70 tx=0x7fec38011810 comp rx=0 tx=0).stop 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec3e000640 1 -- 172.21.15.2:0/1557366851 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fec2c0ef540 con 0x7fec401089b0 2024-03-11T18:33:22.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 -- 172.21.15.2:0/3618574065 shutdown_connections 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x55659292d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 --2- 172.21.15.2:0/3618574065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x5565929259a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec4b188740 1 -- 172.21.15.2:0/3618574065 >> 172.21.15.2:0/3618574065 conn(0x556592920af0 msgr2=0x556592920ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.246+0000 7fec45a00640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 0x7fec240a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.247+0000 7fec4b188740 1 -- 172.21.15.2:0/3618574065 shutdown_connections 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.247+0000 7fec4b188740 1 -- 172.21.15.2:0/3618574065 wait complete. 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.247+0000 7fec45a00640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 0x7fec240a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x556592921a60 tx=0x7fec30009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.247+0000 7fec4b188740 1 -- start start 2024-03-11T18:33:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.247+0000 7fec4b188740 1 Processor -- start 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x55659292f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x556592936990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592832ea0 con 0x55659292b0e0 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592832b40 con 0x556592925f70 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565925d5f20 con 0x556592925580 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec4b188740 10 client.0 _ll_register_callbacks cb 0x556592846950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec46400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x556592936990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec45000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec45000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46140/0 (socket says 172.21.15.2:46140) 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec45a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x55659292f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.248+0000 7fec45000640 1 -- 172.21.15.2:0/3730789358 learned_addr learned my addr 172.21.15.2:0/3730789358 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 -- 172.21.15.2:0/3730789358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 msgr2=0x55659292f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x55659292f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 -- 172.21.15.2:0/3730789358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 msgr2=0x556592936990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x556592936990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 -- 172.21.15.2:0/3730789358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec38009a20 con 0x556592925f70 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec46400640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x556592936990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec45000640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0b85a0 tx=0x7fec2c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d8c51a2d8566214 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec2b400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0ef030 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fec2c0bc140 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.249+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565928b1c40 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec2c0d2050 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 10 client.25331 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55659288bec0 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5565928b1c40 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec2c0c4410 con 0x556592925f70 2024-03-11T18:33:22.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 10 client.25331 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 10 client.25331 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 20 client.25331 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x55659293b5c0 0x55659293fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 4 client.25331 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.250+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 --> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55659258d6c0 con 0x55659293b5c0 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.251+0000 7fec45a00640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x55659293b5c0 0x55659293fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.251+0000 7fec45a00640 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x55659293b5c0 0x55659293fa40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14f978c938ec5dea server_cookie=4fb2e0de12c7f285 in_seq=0 out_seq=0 2024-03-11T18:33:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.251+0000 7fec2b400640 10 client.25331 ms_handle_connect on v2:172.21.15.23:6836/531049865 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec2b400640 1 -- 172.21.15.2:0/3730789358 <== mds.2 v2:172.21.15.23:6836/531049865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55659258d6c0 con 0x55659293b5c0 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec2b400640 10 client.25331 handle_command_reply: tid=0 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 client.25331 shutdown 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 msgr2=0x55659292f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0b85a0 tx=0x7fec2c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 shutdown_connections 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x55659293b5c0 0x55659293fa40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55659292b0e0 0x556592936990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x55659292f950 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec2b400640 0 client.25331 ms_handle_reset on v2:172.21.15.23:6836/531049865 2024-03-11T18:33:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 --2- 172.21.15.2:0/3730789358 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556592925580 0x55659292f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.252+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 >> 172.21.15.2:0/3730789358 conn(0x556592920af0 msgr2=0x55659291fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.253+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 shutdown_connections 2024-03-11T18:33:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.253+0000 7fec4b188740 1 -- 172.21.15.2:0/3730789358 wait complete. 2024-03-11T18:33:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.253+0000 7fec4b188740 20 client.25331 trim_cache size 0 max 16384 2024-03-11T18:33:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.270+0000 7fec4b188740 1 Processor -- start 2024-03-11T18:33:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.270+0000 7fec4b188740 1 -- start start 2024-03-11T18:33:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 0x55659291f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x556592936950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592832b40 con 0x556592925580 2024-03-11T18:33:22.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55659292d700 con 0x556592925f70 2024-03-11T18:33:22.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec4b188740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592937250 con 0x55659292b0e0 2024-03-11T18:33:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec45a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 0x55659291f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec46400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec46400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59618/0 (socket says 172.21.15.2:59618) 2024-03-11T18:33:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec46400640 1 -- 172.21.15.2:0/3493043645 learned_addr learned my addr 172.21.15.2:0/3493043645 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.271+0000 7fec45000640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x556592936950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 -- 172.21.15.2:0/3493043645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 msgr2=0x55659291fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 -- 172.21.15.2:0/3493043645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 msgr2=0x556592936950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x556592936950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 -- 172.21.15.2:0/3493043645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565928288b0 con 0x556592925580 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45000640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x556592936950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec46400640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec45a00640 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 0x55659291f580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec30009fd0 tx=0x7fec30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451613c54480d368 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec36000640 1 -- 172.21.15.2:0/3493043645 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30023040 con 0x556592925580 2024-03-11T18:33:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec36000640 1 -- 172.21.15.2:0/3493043645 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec30028670 con 0x556592925580 2024-03-11T18:33:22.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.272+0000 7fec36000640 1 -- 172.21.15.2:0/3493043645 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3001c620 con 0x556592925580 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 -- 172.21.15.2:0/3493043645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 msgr2=0x55659291f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 0x55659291f580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec30009fd0 tx=0x7fec30016000 comp rx=0 tx=0).stop 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 -- 172.21.15.2:0/3493043645 shutdown_connections 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x55659291fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925f70 0x556592936950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 --2- 172.21.15.2:0/3493043645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925580 0x55659291f580 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 -- 172.21.15.2:0/3493043645 >> 172.21.15.2:0/3493043645 conn(0x556592920af0 msgr2=0x5565929442f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 -- 172.21.15.2:0/3493043645 shutdown_connections 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.273+0000 7fec4b188740 1 -- 172.21.15.2:0/3493043645 wait complete. 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 -- start start 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 Processor -- start 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x556592934f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55659293be50 con 0x556592925f70 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592939830 con 0x556592925580 2024-03-11T18:33:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556592832ea0 con 0x55659292b0e0 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.274+0000 7fec4b188740 10 client.0 _ll_register_callbacks cb 0x7fec2c0097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43050/0 (socket says 172.21.15.2:43050) 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 -- 172.21.15.2:0/2528475767 learned_addr learned my addr 172.21.15.2:0/2528475767 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46166/0 (socket says 172.21.15.2:46166) 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec46400640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x556592934f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 -- 172.21.15.2:0/2528475767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 msgr2=0x556592934f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x556592934f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 -- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 msgr2=0x5565929339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 -- 172.21.15.2:0/2528475767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55659288bec0 con 0x556592925f70 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45a00640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec45000640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0c4300 tx=0x7fec2c0ef3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfb4c6f02c1a7f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.275+0000 7fec2b400640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0d2070 con 0x556592925f70 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fec2c0da5b0 con 0x556592925f70 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55659293bf90 con 0x556592925f70 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec2c0ef040 con 0x556592925f70 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec4b188740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5565928554c0 con 0x556592925f70 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55659293bf90 con 0x556592925f70 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec2c02ca70 con 0x556592925f70 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.276+0000 7fec4b188740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec4b188740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec4b188740 20 client.15240 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x5565929969a0 0x556592936620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec4b188740 4 client.15240 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 --> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fec38004390 con 0x5565929969a0 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec45a00640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x5565929969a0 0x556592936620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec45a00640 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x5565929969a0 0x556592936620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cfe170e373f271fc server_cookie=7a0fa3e1c963cac7 in_seq=0 out_seq=0 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.277+0000 7fec2b400640 10 client.15240 ms_handle_connect on v2:172.21.15.23:6836/531049865 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec2b400640 1 -- 172.21.15.2:0/2528475767 <== mds.2 v2:172.21.15.23:6836/531049865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7fec38004390 con 0x5565929969a0 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec2b400640 10 client.15240 handle_command_reply: tid=0 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 client.15240 shutdown 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 msgr2=0x556592933ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0c4300 tx=0x7fec2c0ef3d0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 shutdown_connections 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:6836/531049865,v1:172.21.15.23:6837/531049865] conn(0x5565929969a0 0x556592936620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55659292b0e0 0x556592934f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556592925f70 0x556592933ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 --2- 172.21.15.2:0/2528475767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556592925580 0x5565929339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec2b400640 0 client.15240 ms_handle_reset on v2:172.21.15.23:6836/531049865 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 >> 172.21.15.2:0/2528475767 conn(0x556592920af0 msgr2=0x556592923d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 shutdown_connections 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 1 -- 172.21.15.2:0/2528475767 wait complete. 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.278+0000 7fec4b188740 20 client.15240 trim_cache size 0 max 16384 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 2, 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24521, 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:33:22.287 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 141.82934769600001, 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 194.38375222799999 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:33:22.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 msgr2=0x7fec240a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 0x7fec240a3ef0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x556592921a60 tx=0x7fec30009720 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 msgr2=0x7fec4019dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fec2c005940 tx=0x7fec2c035f90 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 shutdown_connections 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fec240a1a30 0x7fec240a3ef0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec40109370 0x7fec401a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fec401089b0 0x7fec4019dde0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 --2- 172.21.15.2:0/1557366851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec401077c0 0x7fec4019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 >> 172.21.15.2:0/1557366851 conn(0x7fec40076770 msgr2=0x7fec4010bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 shutdown_connections 2024-03-11T18:33:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.279+0000 7fec48400640 1 -- 172.21.15.2:0/1557366851 wait complete. 2024-03-11T18:33:22.298 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-11T18:33:22.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 -- 172.21.15.2:0/2898285661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 msgr2=0x7fc5a0102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 --2- 172.21.15.2:0/2898285661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0102640 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009900 tx=0x7fc58c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 -- 172.21.15.2:0/2898285661 shutdown_connections 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 --2- 172.21.15.2:0/2898285661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 0x7fc5a0109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 --2- 172.21.15.2:0/2898285661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0102640 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 --2- 172.21.15.2:0/2898285661 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a01013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 -- 172.21.15.2:0/2898285661 >> 172.21.15.2:0/2898285661 conn(0x7fc5a00fc540 msgr2=0x7fc5a00fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.512+0000 7fc59f200640 1 -- 172.21.15.2:0/2898285661 shutdown_connections 2024-03-11T18:33:22.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.513+0000 7fc59f200640 1 -- 172.21.15.2:0/2898285661 wait complete. 2024-03-11T18:33:22.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.513+0000 7fc59f200640 1 Processor -- start 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 -- start start 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 0x7fc5a0073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0074210 con 0x7fc5a01021c0 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0074380 con 0x7fc5a0102b80 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a01a10a0 con 0x7fc5a0100fd0 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59632/0 (socket says 172.21.15.2:59632) 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59de00640 1 -- 172.21.15.2:0/2393747073 learned_addr learned my addr 172.21.15.2:0/2393747073 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59d400640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.514+0000 7fc59e800640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 0x7fc5a0073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 -- 172.21.15.2:0/2393747073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 msgr2=0x7fc5a0073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 0x7fc5a0073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 -- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 msgr2=0x7fc5a0073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 -- 172.21.15.2:0/2393747073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c009560 con 0x7fc5a0100fd0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59d400640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59de00640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc59000e9c0 tx=0x7fc59000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc596a00640 1 -- 172.21.15.2:0/2393747073 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590009700 con 0x7fc5a0100fd0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc596a00640 1 -- 172.21.15.2:0/2393747073 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc59000cd80 con 0x7fc5a0100fd0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc596a00640 1 -- 172.21.15.2:0/2393747073 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590010430 con 0x7fc5a0100fd0 2024-03-11T18:33:22.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.515+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a01a1350 con 0x7fc5a0100fd0 2024-03-11T18:33:22.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.516+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a01a17e0 con 0x7fc5a0100fd0 2024-03-11T18:33:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.520+0000 7fc5a7dea740 1 Processor -- start 2024-03-11T18:33:22.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.520+0000 7fc5a7dea740 1 -- start start 2024-03-11T18:33:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.520+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1a6150 0x559a4b1aac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b114b30 con 0x559a4b1a5760 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc596a00640 1 -- 172.21.15.2:0/2393747073 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fc590003680 con 0x7fc5a0100fd0 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b112060 con 0x559a4b1a6150 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b10d840 con 0x559a4b1ab2c0 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1a6150 0x559a4b1aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59638/0 (socket says 172.21.15.2:59638) 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 -- 172.21.15.2:0/3885768876 learned_addr learned my addr 172.21.15.2:0/3885768876 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59e800640 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 -- 172.21.15.2:0/3885768876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1a6150 msgr2=0x559a4b1aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1a6150 0x559a4b1aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 -- 172.21.15.2:0/3885768876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 msgr2=0x559a4b1a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.521+0000 7fc59d400640 1 -- 172.21.15.2:0/3885768876 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a4b0d54c0 con 0x559a4b1ab2c0 2024-03-11T18:33:22.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.522+0000 7fc59d400640 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c02c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=737eee35d6983c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.522+0000 7fc58b400640 1 -- 172.21.15.2:0/3885768876 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0046c0 con 0x559a4b1ab2c0 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.522+0000 7fc58b400640 1 -- 172.21.15.2:0/3885768876 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc58c004820 con 0x559a4b1ab2c0 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.522+0000 7fc58b400640 1 -- 172.21.15.2:0/3885768876 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c005570 con 0x559a4b1ab2c0 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc596a00640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 0x7fc5840a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc596a00640 1 -- 172.21.15.2:0/2393747073 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc590014070 con 0x7fc5a0100fd0 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc59e800640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 0x7fc5840a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 -- 172.21.15.2:0/3885768876 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 msgr2=0x559a4b1ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c02c8e0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 -- 172.21.15.2:0/3885768876 shutdown_connections 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1ab2c0 0x559a4b1ad750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1a6150 0x559a4b1aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/3885768876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.523+0000 7fc5a7dea740 1 -- 172.21.15.2:0/3885768876 >> 172.21.15.2:0/3885768876 conn(0x559a4b1a0cd0 msgr2=0x559a4b1a10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.524+0000 7fc5a7dea740 1 -- 172.21.15.2:0/3885768876 shutdown_connections 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.524+0000 7fc5a7dea740 1 -- 172.21.15.2:0/3885768876 wait complete. 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.524+0000 7fc59e800640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 0x7fc5840a3f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x559a4b1a69b0 tx=0x7fc598009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.524+0000 7fc5a7dea740 1 -- start start 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.524+0000 7fc5a7dea740 1 Processor -- start 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b1af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b0b2ea0 con 0x559a4b1a6150 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b0b2b40 con 0x559a4b1ab2c0 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4ae55f20 con 0x559a4b1a5760 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc5a7dea740 10 client.0 _ll_register_callbacks cb 0x559a4b0c6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc59d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc59e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc59d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43076/0 (socket says 172.21.15.2:43076) 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.525+0000 7fc59d400640 1 -- 172.21.15.2:0/1201171960 learned_addr learned my addr 172.21.15.2:0/1201171960 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59de00640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b1af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 -- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 msgr2=0x559a4b1af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b1af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 -- 172.21.15.2:0/1201171960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 msgr2=0x559a4b1b6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 -- 172.21.15.2:0/1201171960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a0109d10 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59e800640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59de00640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b1af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc59d400640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005ec0 tx=0x7fc58c02c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b042cab2a10ab116 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc57fe00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c049450 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.526+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc58c0495b0 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a4b131c40 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc58c049770 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 10 client.15252 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559a4b10bec0 con 0x559a4b1a6150 2024-03-11T18:33:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559a4b131c40 con 0x559a4b1a6150 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc58c03c040 con 0x559a4b1a6150 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 10 client.15252 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 10 client.15252 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.527+0000 7fc5a7dea740 20 client.15252 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b1bb0e0 0x559a4b1bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc5a7dea740 4 client.15252 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 --> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a4ae0d6c0 con 0x559a4b1bb0e0 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc59de00640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b1bb0e0 0x559a4b1bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc59de00640 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b1bb0e0 0x559a4b1bfb80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96809047cb00f2bc server_cookie=3a693464b8399369 in_seq=0 out_seq=0 2024-03-11T18:33:22.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.528+0000 7fc57fe00640 10 client.15252 ms_handle_connect on v2:172.21.15.165:6838/1329073865 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc57fe00640 1 -- 172.21.15.2:0/1201171960 <== mds.1 v2:172.21.15.165:6838/1329073865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559a4ae0d6c0 con 0x559a4b1bb0e0 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc57fe00640 10 client.15252 handle_command_reply: tid=0 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 client.15252 shutdown 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 msgr2=0x559a4b1afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc58c005ec0 tx=0x7fc58c02c350 comp rx=0 tx=0).stop 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 shutdown_connections 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b1bb0e0 0x559a4b1bfb80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1afaa0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1201171960 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b1af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 >> 172.21.15.2:0/1201171960 conn(0x559a4b1a0cd0 msgr2=0x559a4b19fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.529+0000 7fc57fe00640 0 client.15252 ms_handle_reset on v2:172.21.15.165:6838/1329073865 2024-03-11T18:33:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.530+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 shutdown_connections 2024-03-11T18:33:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.530+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1201171960 wait complete. 2024-03-11T18:33:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.530+0000 7fc5a7dea740 20 client.15252 trim_cache size 0 max 16384 2024-03-11T18:33:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.550+0000 7fc5a7dea740 1 Processor -- start 2024-03-11T18:33:22.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 -- start start 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b1ade10 con 0x559a4b1a6150 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4ae0d6c0 con 0x559a4b1ab2c0 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b131c40 con 0x559a4b1a5760 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59654/0 (socket says 172.21.15.2:59654) 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43080/0 (socket says 172.21.15.2:43080) 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59de00640 1 -- 172.21.15.2:0/2296546838 learned_addr learned my addr 172.21.15.2:0/2296546838 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.551+0000 7fc59e800640 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 -- 172.21.15.2:0/2296546838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 msgr2=0x559a4b1c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1c3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 -- 172.21.15.2:0/2296546838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 msgr2=0x559a4b1b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 -- 172.21.15.2:0/2296546838 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a4b0a88b0 con 0x559a4b1a5760 2024-03-11T18:33:22.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59d400640 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc59de00640 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc590097180 tx=0x7fc5900cb210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef57011734b8dd49 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc58aa00640 1 -- 172.21.15.2:0/2296546838 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59009a580 con 0x559a4b1a5760 2024-03-11T18:33:22.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc58aa00640 1 -- 172.21.15.2:0/2296546838 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc59009a2f0 con 0x559a4b1a5760 2024-03-11T18:33:22.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.552+0000 7fc58aa00640 1 -- 172.21.15.2:0/2296546838 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5900a2f50 con 0x559a4b1a5760 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.553+0000 7fc5a7dea740 1 -- 172.21.15.2:0/2296546838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 msgr2=0x559a4b19fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.553+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc590097180 tx=0x7fc5900cb210 comp rx=0 tx=0).stop 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 -- 172.21.15.2:0/2296546838 shutdown_connections 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1c3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a6150 0x559a4b1b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/2296546838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a5760 0x559a4b19fc90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 -- 172.21.15.2:0/2296546838 >> 172.21.15.2:0/2296546838 conn(0x559a4b1a0cd0 msgr2=0x559a4b19f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 -- 172.21.15.2:0/2296546838 shutdown_connections 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 -- 172.21.15.2:0/2296546838 wait complete. 2024-03-11T18:33:22.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.554+0000 7fc5a7dea740 1 -- start start 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 Processor -- start 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a6150 0x559a4b1b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b1b34b0 con 0x559a4b1a5760 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b1b3620 con 0x559a4b1ab2c0 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a4b1b37e0 con 0x559a4b1a6150 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc5a7dea740 10 client.0 _ll_register_callbacks cb 0x7fc5900d3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc59e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc59de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc59d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a6150 0x559a4b1b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc59e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46232/0 (socket says 172.21.15.2:46232) 2024-03-11T18:33:22.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.555+0000 7fc59e800640 1 -- 172.21.15.2:0/1947599211 learned_addr learned my addr 172.21.15.2:0/1947599211 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 -- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a6150 msgr2=0x559a4b1b68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a6150 0x559a4b1b68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 -- 172.21.15.2:0/1947599211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 msgr2=0x559a4b1b63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1b63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 -- 172.21.15.2:0/1947599211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a4b10bec0 con 0x559a4b1ab2c0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc59e800640 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc598012c40 tx=0x7fc598004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa04fe29b3a58b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc57fe00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.556+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598022070 con 0x559a4b1ab2c0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a4b1bba70 con 0x559a4b1ab2c0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc598004d20 con 0x559a4b1ab2c0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc598005020 con 0x559a4b1ab2c0 2024-03-11T18:33:22.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc5a7dea740 10 client.25355 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fc5a01a17e0 con 0x559a4b1ab2c0 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559a4b1bba70 con 0x559a4b1ab2c0 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc598027700 con 0x559a4b1ab2c0 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.557+0000 7fc5a7dea740 10 client.25355 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc5a7dea740 10 client.25355 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc5a7dea740 20 client.25355 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b2167f0 0x559a4b212c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc5a7dea740 4 client.25355 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:33:22.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 --> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fc58c004820 con 0x559a4b2167f0 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc59de00640 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b2167f0 0x559a4b212c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc59de00640 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b2167f0 0x559a4b212c90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2bfdfcabafb77c7f server_cookie=136eab0b3f84a969 in_seq=0 out_seq=0 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.558+0000 7fc57fe00640 10 client.25355 ms_handle_connect on v2:172.21.15.165:6838/1329073865 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc57fe00640 1 -- 172.21.15.2:0/1947599211 <== mds.1 v2:172.21.15.165:6838/1329073865 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7fc58c004820 con 0x559a4b2167f0 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc57fe00640 10 client.25355 handle_command_reply: tid=0 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 client.25355 shutdown 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 msgr2=0x559a4b1b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc598012c40 tx=0x7fc598004a40 comp rx=0 tx=0).stop 2024-03-11T18:33:22.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 shutdown_connections 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:6838/1329073865,v1:172.21.15.165:6839/1329073865] conn(0x559a4b2167f0 0x559a4b212c90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559a4b1ab2c0 0x559a4b1b2ea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559a4b1a6150 0x559a4b1b68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 --2- 172.21.15.2:0/1947599211 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559a4b1a5760 0x559a4b1b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc57fe00640 0 client.25355 ms_handle_reset on v2:172.21.15.165:6838/1329073865 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 >> 172.21.15.2:0/1947599211 conn(0x559a4b1a0cd0 msgr2=0x559a4b19fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 shutdown_connections 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.559+0000 7fc5a7dea740 1 -- 172.21.15.2:0/1947599211 wait complete. 2024-03-11T18:33:22.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.560+0000 7fc5a7dea740 20 client.25355 trim_cache size 0 max 16384 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 1, 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24550, 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 143.137348645, 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:33:22.566 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:33:22.567 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:33:22.567 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 191.717942986 2024-03-11T18:33:22.567 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:33:22.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.560+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 msgr2=0x7fc5840a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.560+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 0x7fc5840a3f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x559a4b1a69b0 tx=0x7fc598009500 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.560+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 msgr2=0x7fc5a0074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.560+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc59000e9c0 tx=0x7fc59000ee80 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 shutdown_connections 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc5840a1ab0 0x7fc5840a3f70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5a0102b80 0x7fc5a0073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5a01021c0 0x7fc5a0073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 --2- 172.21.15.2:0/2393747073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a0100fd0 0x7fc5a0074f30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 >> 172.21.15.2:0/2393747073 conn(0x7fc5a00fc540 msgr2=0x7fc5a01054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 shutdown_connections 2024-03-11T18:33:22.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.561+0000 7fc59f200640 1 -- 172.21.15.2:0/2393747073 wait complete. 2024-03-11T18:33:22.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:22 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3840765903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:22 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1701735908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.583 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-11T18:33:22.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:22 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3840765903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:22 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1701735908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:22 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3840765903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:22 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1701735908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:33:22.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 -- 172.21.15.2:0/971489677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68881043d0 msgr2=0x7f6888104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 --2- 172.21.15.2:0/971489677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68881043d0 0x7f6888104850 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6878009900 tx=0x7f687802bc40 comp rx=0 tx=0).stop 2024-03-11T18:33:22.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 -- 172.21.15.2:0/971489677 shutdown_connections 2024-03-11T18:33:22.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 --2- 172.21.15.2:0/971489677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6888104d90 0x7f688810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 --2- 172.21.15.2:0/971489677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68881043d0 0x7f6888104850 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 --2- 172.21.15.2:0/971489677 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881031e0 0x7f6888103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.771+0000 7f688d200640 1 -- 172.21.15.2:0/971489677 >> 172.21.15.2:0/971489677 conn(0x7f68880fe750 msgr2=0x7f6888100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.772+0000 7f688d200640 1 -- 172.21.15.2:0/971489677 shutdown_connections 2024-03-11T18:33:22.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.772+0000 7f688d200640 1 -- 172.21.15.2:0/971489677 wait complete. 2024-03-11T18:33:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.772+0000 7f688d200640 1 Processor -- start 2024-03-11T18:33:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 -- start start 2024-03-11T18:33:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68881031e0 0x7f6888199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6888104d90 0x7f68881a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881a0d30 con 0x7f68881031e0 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881a0ea0 con 0x7f6888104d90 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f688d200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68881a1060 con 0x7f68881043d0 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f6887400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68881031e0 0x7f6888199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f6886a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f6887e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6888104d90 0x7f68881a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f6886a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59664/0 (socket says 172.21.15.2:59664) 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.773+0000 7f6886a00640 1 -- 172.21.15.2:0/186992614 learned_addr learned my addr 172.21.15.2:0/186992614 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 -- 172.21.15.2:0/186992614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6888104d90 msgr2=0x7f68881a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6888104d90 0x7f68881a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 -- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68881031e0 msgr2=0x7f6888199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68881031e0 0x7f6888199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 -- 172.21.15.2:0/186992614 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878009560 con 0x7f68881043d0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.774+0000 7f6886a00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f687802c6b0 tx=0x7f68780040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.775+0000 7f687fe00640 1 -- 172.21.15.2:0/186992614 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780044c0 con 0x7f68881043d0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.775+0000 7f687fe00640 1 -- 172.21.15.2:0/186992614 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68780052d0 con 0x7f68881043d0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.775+0000 7f687fe00640 1 -- 172.21.15.2:0/186992614 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878003e80 con 0x7f68881043d0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.775+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68881a12e0 con 0x7f68881043d0 2024-03-11T18:33:22.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.775+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68881a17a0 con 0x7f68881043d0 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.778+0000 7f6891408740 1 Processor -- start 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.778+0000 7f6891408740 1 -- start start 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7c0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb727b30 con 0x5562cb7b8760 2024-03-11T18:33:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb725060 con 0x5562cb7be2c0 2024-03-11T18:33:22.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6891408740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb720840 con 0x5562cb7b9150 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59674/0 (socket says 172.21.15.2:59674) 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 -- 172.21.15.2:0/3072240630 learned_addr learned my addr 172.21.15.2:0/3072240630 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 -- 172.21.15.2:0/3072240630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 msgr2=0x5562cb7c0750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7c0750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 -- 172.21.15.2:0/3072240630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 msgr2=0x5562cb7b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f6887400640 1 -- 172.21.15.2:0/3072240630 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562cb6e84c0 con 0x5562cb7b9150 2024-03-11T18:33:22.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.779+0000 7f687fe00640 1 -- 172.21.15.2:0/186992614 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f6878005440 con 0x7f68881043d0 2024-03-11T18:33:22.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.780+0000 7f6887400640 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6870009ca0 tx=0x7f6870011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1fecaae35646e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.780+0000 7f687cc00640 1 -- 172.21.15.2:0/3072240630 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870004220 con 0x5562cb7b9150 2024-03-11T18:33:22.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.780+0000 7f687cc00640 1 -- 172.21.15.2:0/3072240630 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6870004380 con 0x5562cb7b9150 2024-03-11T18:33:22.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.780+0000 7f687cc00640 1 -- 172.21.15.2:0/3072240630 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870019830 con 0x5562cb7b9150 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 -- 172.21.15.2:0/3072240630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 msgr2=0x5562cb7bdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6870009ca0 tx=0x7f6870011810 comp rx=0 tx=0).stop 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 -- 172.21.15.2:0/3072240630 shutdown_connections 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7c0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7bdc50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 --2- 172.21.15.2:0/3072240630 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 -- 172.21.15.2:0/3072240630 >> 172.21.15.2:0/3072240630 conn(0x5562cb7b3cd0 msgr2=0x5562cb7b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f687fe00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 0x7f686c0a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f687fe00640 1 -- 172.21.15.2:0/186992614 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f68780f00f0 con 0x7f68881043d0 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 -- 172.21.15.2:0/3072240630 shutdown_connections 2024-03-11T18:33:22.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.781+0000 7f6891408740 1 -- 172.21.15.2:0/3072240630 wait complete. 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.782+0000 7f6887e00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 0x7f686c0a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.782+0000 7f6891408740 1 -- start start 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.782+0000 7f6891408740 1 Processor -- start 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.782+0000 7f6887e00640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 0x7f686c0a3f70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5562cb7b99b0 tx=0x7f6880009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 0x5562cb7c2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb6c5ea0 con 0x5562cb7b9150 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb6c5b40 con 0x5562cb7b8760 2024-03-11T18:33:22.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb468f20 con 0x5562cb7be2c0 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6891408740 10 client.0 _ll_register_callbacks cb 0x5562cb6d9950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6887400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6887400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46250/0 (socket says 172.21.15.2:46250) 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6886a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 0x5562cb7c2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6887400640 1 -- 172.21.15.2:0/4250843788 learned_addr learned my addr 172.21.15.2:0/4250843788 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.783+0000 7f6887e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 -- 172.21.15.2:0/4250843788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 msgr2=0x5562cb7c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 -- 172.21.15.2:0/4250843788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 msgr2=0x5562cb7c2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 0x5562cb7c2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 -- 172.21.15.2:0/4250843788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700098d0 con 0x5562cb7b8760 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6886a00640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 0x5562cb7c2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6887400640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6870009870 tx=0x7f6870003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af248bce541dd609 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6875600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870003c60 con 0x5562cb7b8760 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f6870005c50 con 0x5562cb7b8760 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.784+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562cb744c40 con 0x5562cb7b8760 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6870005050 con 0x5562cb7b8760 2024-03-11T18:33:22.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 10 client.25364 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5562cb71eec0 con 0x5562cb7b8760 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5562cb744c40 con 0x5562cb7b8760 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6870019530 con 0x5562cb7b8760 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 10 client.25364 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 10 client.25364 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 20 client.25364 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.785+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb7ce4b0 0x5562cb7d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.786+0000 7f6891408740 4 client.25364 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.786+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562cb4206c0 con 0x5562cb7ce4b0 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.786+0000 7f6886a00640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb7ce4b0 0x5562cb7d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.786+0000 7f6886a00640 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb7ce4b0 0x5562cb7d2f70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ab5503060a7617 server_cookie=1d0d1d631bd4c8b7 in_seq=0 out_seq=0 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.786+0000 7f6875600640 10 client.25364 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6875600640 1 -- 172.21.15.2:0/4250843788 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5562cb4206c0 con 0x5562cb7ce4b0 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6875600640 10 client.25364 handle_command_reply: tid=0 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 client.25364 shutdown 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 msgr2=0x5562cb7c2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6870009870 tx=0x7f6870003820 comp rx=0 tx=0).stop 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 shutdown_connections 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb7ce4b0 0x5562cb7d2f70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b9150 0x5562cb7c2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 --2- 172.21.15.2:0/4250843788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b8760 0x5562cb7c2620 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6875600640 0 client.25364 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:22.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 >> 172.21.15.2:0/4250843788 conn(0x5562cb7b3cd0 msgr2=0x5562cb7b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 shutdown_connections 2024-03-11T18:33:22.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.787+0000 7f6891408740 1 -- 172.21.15.2:0/4250843788 wait complete. 2024-03-11T18:33:22.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.788+0000 7f6891408740 20 client.25364 trim_cache size 0 max 16384 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.805+0000 7f6891408740 1 Processor -- start 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.805+0000 7f6891408740 1 -- start start 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.805+0000 7f6891408740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7d6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6891408740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6891408740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6891408740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7b8410 con 0x5562cb7b8760 2024-03-11T18:33:22.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6891408740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7b4cc0 con 0x5562cb7be2c0 2024-03-11T18:33:22.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6891408740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7af310 con 0x5562cb7b9150 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6887e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6887400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7d6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59694/0 (socket says 172.21.15.2:59694) 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 -- 172.21.15.2:0/3695469271 learned_addr learned my addr 172.21.15.2:0/3695469271 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 -- 172.21.15.2:0/3695469271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 msgr2=0x5562cb7d74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7d74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 -- 172.21.15.2:0/3695469271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 msgr2=0x5562cb7d6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7d6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.806+0000 7f6886a00640 1 -- 172.21.15.2:0/3695469271 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562cb6bb8b0 con 0x5562cb7b9150 2024-03-11T18:33:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.807+0000 7f6886a00640 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68780b9220 tx=0x7f6878041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d583410262387db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.807+0000 7f6877e00640 1 -- 172.21.15.2:0/3695469271 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780c5560 con 0x5562cb7b9150 2024-03-11T18:33:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.807+0000 7f6877e00640 1 -- 172.21.15.2:0/3695469271 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68780bae60 con 0x5562cb7b9150 2024-03-11T18:33:22.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.807+0000 7f6877e00640 1 -- 172.21.15.2:0/3695469271 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780c50d0 con 0x5562cb7b9150 2024-03-11T18:33:22.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- 172.21.15.2:0/3695469271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 msgr2=0x5562cb7c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68780b9220 tx=0x7f6878041040 comp rx=0 tx=0).stop 2024-03-11T18:33:22.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- 172.21.15.2:0/3695469271 shutdown_connections 2024-03-11T18:33:22.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7be2c0 0x5562cb7d74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7b9150 0x5562cb7c9b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 --2- 172.21.15.2:0/3695469271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7d6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- 172.21.15.2:0/3695469271 >> 172.21.15.2:0/3695469271 conn(0x5562cb7b3cd0 msgr2=0x5562cb7b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- 172.21.15.2:0/3695469271 shutdown_connections 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- 172.21.15.2:0/3695469271 wait complete. 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.808+0000 7f6891408740 1 -- start start 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 Processor -- start 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 0x5562cb7c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7c67c0 con 0x5562cb7b8760 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7c6930 con 0x5562cb7b9150 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562cb7c6af0 con 0x5562cb7be2c0 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6891408740 10 client.0 _ll_register_callbacks cb 0x7f6880009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6886a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 0x5562cb7c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6887e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6887400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6886a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 0x5562cb7c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46268/0 (socket says 172.21.15.2:46268) 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6887e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59696/0 (socket says 172.21.15.2:59696) 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.809+0000 7f6886a00640 1 -- 172.21.15.2:0/1330895129 learned_addr learned my addr 172.21.15.2:0/1330895129 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 -- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 msgr2=0x5562cb7c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 0x5562cb7c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 -- 172.21.15.2:0/1330895129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 msgr2=0x5562cb7c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 -- 172.21.15.2:0/1330895129 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68780c0040 con 0x5562cb7be2c0 2024-03-11T18:33:22.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6887e00640 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5562cb7ca160 tx=0x7f6880004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac3c141987a999bb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6875600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6880024070 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562cb7cee60 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.810+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f688000fbe0 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6880016070 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 10 client.25375 fetch_fsmap learned FSMap version 24 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5562cb6e1510 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5562cb7cee60 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f688001e400 con 0x5562cb7be2c0 2024-03-11T18:33:22.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 10 client.25375 fetch_fsmap finished waiting for FSMap version 24 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 10 client.25375 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 20 client.25375 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb829d50 0x5562cb8261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 4 client.25375 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6870004380 con 0x5562cb829d50 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.811+0000 7f6887400640 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb829d50 0x5562cb8261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6887400640 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb829d50 0x5562cb8261b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73050b92d4634fb8 server_cookie=c84c9fc39bea96f5 in_seq=0 out_seq=0 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6875600640 10 client.25375 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6875600640 1 -- 172.21.15.2:0/1330895129 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x7f6870004380 con 0x5562cb829d50 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6875600640 10 client.25375 handle_command_reply: tid=0 2024-03-11T18:33:22.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 client.25375 shutdown 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 msgr2=0x5562cb7c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5562cb7ca160 tx=0x7f6880004a40 comp rx=0 tx=0).stop 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 shutdown_connections 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5562cb829d50 0x5562cb8261b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5562cb7be2c0 0x5562cb7c60c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5562cb7b9150 0x5562cb7c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 --2- 172.21.15.2:0/1330895129 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5562cb7b8760 0x5562cb7c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6875600640 0 client.25375 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.812+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 >> 172.21.15.2:0/1330895129 conn(0x5562cb7b3cd0 msgr2=0x5562cb7d6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 shutdown_connections 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f6891408740 1 -- 172.21.15.2:0/1330895129 wait complete. 2024-03-11T18:33:22.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f6891408740 20 client.25375 trim_cache size 0 max 16384 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "02b83714-dfd4-11ee-95c8-87774f69a715", 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 24575, 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 174.54541553499999, 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 24, 2024-03-11T18:33:22.824 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 95, 2024-03-11T18:33:22.825 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 91, 2024-03-11T18:33:22.825 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 181.02124113299999 2024-03-11T18:33:22.825 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:33:22.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 msgr2=0x7f686c0a3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 0x7f686c0a3f70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5562cb7b99b0 tx=0x7f6880009500 comp rx=0 tx=0).stop 2024-03-11T18:33:22.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 msgr2=0x7f6888199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:22.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f687802c6b0 tx=0x7f68780040c0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 shutdown_connections 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f686c0a1ab0 0x7f686c0a3f70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6888104d90 0x7f68881a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68881043d0 0x7f6888199780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 --2- 172.21.15.2:0/186992614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68881031e0 0x7f6888199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.813+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 >> 172.21.15.2:0/186992614 conn(0x7f68880fe750 msgr2=0x7f6888107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.814+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 shutdown_connections 2024-03-11T18:33:22.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:22.814+0000 7f688d200640 1 -- 172.21.15.2:0/186992614 wait complete. 2024-03-11T18:33:22.834 INFO:teuthology.run_tasks:Running task kclient... 2024-03-11T18:33:22.848 INFO:tasks.kclient:Mounting kernel clients... 2024-03-11T18:33:22.848 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-11T18:33:22.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:22.848 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-03-11T18:33:22.899 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-11T18:33:22.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:22.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-11T18:33:22.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:22.987 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2024-03-11T18:33:23.113 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-11T18:33:23.114 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T18:33:23.114 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:23.114 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T18:33:23.582 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:33:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: pgmap v412: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:23 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: pgmap v412: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:23 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: pgmap v412: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:23.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:23 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:24.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:24 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:24.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:24 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:24.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:24 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:25.845 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:25 smithi002 ceph-mon[31328]: pgmap v413: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:25.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:25.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:25.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:25.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:25 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 -- 172.21.15.2:0/630385460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c1028d0 msgr2=0x7f312c106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 --2- 172.21.15.2:0/630385460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c1028d0 0x7f312c106910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3118009990 tx=0x7f3118009ca0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 -- 172.21.15.2:0/630385460 shutdown_connections 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 --2- 172.21.15.2:0/630385460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c106ff0 0x7f312c10ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 --2- 172.21.15.2:0/630385460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c101f20 0x7f312c102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 --2- 172.21.15.2:0/630385460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c1028d0 0x7f312c106910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.054+0000 7f3131c00700 1 -- 172.21.15.2:0/630385460 >> 172.21.15.2:0/630385460 conn(0x7f312c078dd0 msgr2=0x7f312c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.055+0000 7f3131c00700 1 -- 172.21.15.2:0/630385460 shutdown_connections 2024-03-11T18:33:26.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.055+0000 7f3131c00700 1 -- 172.21.15.2:0/630385460 wait complete. 2024-03-11T18:33:26.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 Processor -- start 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 -- start start 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 0x7f312c0744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c106ff0 0x7f312c0749f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.056+0000 7f3131c00700 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1a6650 con 0x7f312c106ff0 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f3131c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1a67c0 con 0x7f312c1028d0 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f3131c00700 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c1a6980 con 0x7f312c101f20 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 0x7f312c0744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 0x7f312c0744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46278/0 (socket says 172.21.15.2:46278) 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312a000700 1 -- 172.21.15.2:0/1691132740 learned_addr learned my addr 172.21.15.2:0/1691132740 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:59714/0 (socket says 172.21.15.2:59714) 2024-03-11T18:33:26.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312b400700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c106ff0 0x7f312c0749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:26.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 -- 172.21.15.2:0/1691132740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 msgr2=0x7f312c0744b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 0x7f312c0744b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 -- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c106ff0 msgr2=0x7f312c0749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c106ff0 0x7f312c0749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.057+0000 7f312aa00700 1 -- 172.21.15.2:0/1691132740 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180095f0 con 0x7f312c101f20 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f312aa00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e9f0 tx=0x7f311c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c00c9c0 con 0x7f312c101f20 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f311c010430 con 0x7f312c101f20 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c1a6c60 con 0x7f312c101f20 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c018680 con 0x7f312c101f20 2024-03-11T18:33:26.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.058+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c1a71b0 con 0x7f312c101f20 2024-03-11T18:33:26.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.059+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c10a310 con 0x7f312c101f20 2024-03-11T18:33:26.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.062+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f311c010910 con 0x7f312c101f20 2024-03-11T18:33:26.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.063+0000 7f3123400700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 0x7f31100a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:26.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.063+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f311c014070 con 0x7f312c101f20 2024-03-11T18:33:26.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.063+0000 7f312a000700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 0x7f31100a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:26.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.064+0000 7f312a000700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 0x7f31100a4000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f312c101830 tx=0x7f3118005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:26.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.067+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f311c052d20 con 0x7f312c101f20 2024-03-11T18:33:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:25 smithi165 ceph-mon[36634]: pgmap v413: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:25 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:25 smithi023 ceph-mon[36406]: pgmap v413: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:25 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:26.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.346+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f312c068930 con 0x7f312c101f20 2024-03-11T18:33:26.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.346+0000 7f3123400700 1 -- 172.21.15.2:0/1691132740 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7f311c0966e0 con 0x7f312c101f20 2024-03-11T18:33:26.348 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 msgr2=0x7f31100a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 0x7f31100a4000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f312c101830 tx=0x7f3118005fd0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 msgr2=0x7f312c073f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e9f0 tx=0x7f311c00ed00 comp rx=0 tx=0).stop 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 shutdown_connections 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f31100a1b40 0x7f31100a4000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f312c101f20 0x7f312c073f70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f312c106ff0 0x7f312c0749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 --2- 172.21.15.2:0/1691132740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c1028d0 0x7f312c0744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:26.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 >> 172.21.15.2:0/1691132740 conn(0x7f312c078dd0 msgr2=0x7f312c0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:26.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 shutdown_connections 2024-03-11T18:33:26.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:26.350+0000 7f3131c00700 1 -- 172.21.15.2:0/1691132740 wait complete. 2024-03-11T18:33:26.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:26 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1691132740' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:33:27.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:26 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1691132740' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:33:27.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:26 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1691132740' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T18:33:27.239 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T18:33:27.240 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-11T18:33:27.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.241 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:33:27.293 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41905sec preferred_lft 41905sec 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-03-11T18:33:27.294 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-03-11T18:33:27.295 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T18:33:27.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T18:33:27.296 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:33:27.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.493 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T18:33:27.570 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-03-11T18:33:27.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.573 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-03-11T18:33:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-03-11T18:33:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T18:33:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-11T18:33:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-11T18:33:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T18:33:27.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.620 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:33:27.620 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:33:27.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-11T18:33:27.620 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-11T18:33:27.621 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-11T18:33:27.621 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:33:27.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.829 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-03-11T18:33:27.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.869 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2024-03-11T18:33:27.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:27.954 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:33:27.954 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:33:27.954 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T18:33:27.954 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-11T18:33:27.955 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:33:28.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:27 smithi002 ceph-mon[31328]: pgmap v414: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:28.127 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-11T18:33:28.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-11T18:33:28.128 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:33:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:27 smithi165 ceph-mon[36634]: pgmap v414: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:27 smithi023 ceph-mon[36406]: pgmap v414: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-11T18:33:28.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:28.370 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-03-11T18:33:28.370 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-03-11T18:33:28.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-03-11T18:33:28.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T18:33:28.371 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-03-11T18:33:28.535 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T18:33:28.536 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:28.581 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-11T18:33:28.581 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T18:33:28.581 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:28.671 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-11T18:33:28.671 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-11T18:33:28.671 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/9e52e265-fb66-4d5f-95fb-56794f10b175 2024-03-11T18:33:28.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:28.672 DEBUG:teuthology.orchestra.run.smithi002:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/9e52e265-fb66-4d5f-95fb-56794f10b175 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T18:33:28.786 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T18:33:28.787 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-11T18:33:28.787 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2024-03-11T18:33:29.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:29 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:29 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:29 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:invalid new device string format 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: resolved to: "172.21.15.2:3300,172.21.15.23:3300,172.21.15.165:3300" 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.2:3300,172.21.15.23:3300,172.21.15.165:3300:/volumes/qa/sv_1/9e52e265-fb66-4d5f-95fb-56794f10b175 2024-03-11T18:33:29.544 INFO:teuthology.orchestra.run.smithi002.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=02b83714-dfd4-11ee-95c8-87774f69a715" will pass to kernel 2024-03-11T18:33:29.550 INFO:teuthology.orchestra.run.smithi002.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-11T18:33:29.550 INFO:teuthology.orchestra.run.smithi002.stdout: You just mounted a file system that supports labels which does not 2024-03-11T18:33:29.550 INFO:teuthology.orchestra.run.smithi002.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-11T18:33:29.550 INFO:teuthology.orchestra.run.smithi002.stdout: applications will generate AVC messages and not be allowed access to 2024-03-11T18:33:29.550 INFO:teuthology.orchestra.run.smithi002.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-11T18:33:29.556 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-11T18:33:29.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:29.556 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:29.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:29.656 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T18:33:29.718 INFO:teuthology.orchestra.run.smithi002.stdout:client25379 2024-03-11T18:33:29.718 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:33:29.718 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T18:33:29.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:29.771 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T18:33:29.827 INFO:teuthology.orchestra.run.smithi002.stdout:client25379 2024-03-11T18:33:29.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:29.828 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T18:33:29.887 INFO:teuthology.orchestra.run.smithi002.stdout:client25379 2024-03-11T18:33:29.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:33:29.888 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/02b83714-dfd4-11ee-95c8-87774f69a715.client25379/status') 2024-03-11T18:33:29.989 INFO:teuthology.orchestra.run.smithi002.stdout:instance: client.25379 (3)172.21.15.2:0/3929934130 2024-03-11T18:33:29.989 INFO:teuthology.orchestra.run.smithi002.stdout:blocklisted: false 2024-03-11T18:33:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records in 2024-03-11T18:33:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:0+1 records out 2024-03-11T18:33:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:70 bytes copied, 7.8557e-05 s, 891 kB/s 2024-03-11T18:33:29.994 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T18:33:30.005 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T18:33:30.005 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:33:30.006 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 -- 172.21.15.2:0/1967657095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 msgr2=0x7fa59c102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 --2- 172.21.15.2:0/1967657095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c102830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa588009900 tx=0x7fa58802bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 -- 172.21.15.2:0/1967657095 shutdown_connections 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 --2- 172.21.15.2:0/1967657095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 --2- 172.21.15.2:0/1967657095 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c102830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 --2- 172.21.15.2:0/1967657095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 0x7fa59c1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 -- 172.21.15.2:0/1967657095 >> 172.21.15.2:0/1967657095 conn(0x7fa59c0fc750 msgr2=0x7fa59c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 -- 172.21.15.2:0/1967657095 shutdown_connections 2024-03-11T18:33:30.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.230+0000 7fa5a4600640 1 -- 172.21.15.2:0/1967657095 wait complete. 2024-03-11T18:33:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.231+0000 7fa5a4600640 1 Processor -- start 2024-03-11T18:33:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.231+0000 7fa5a4600640 1 -- start start 2024-03-11T18:33:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 0x7fa59c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:30.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c1a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1a0f80 con 0x7fa59c102d70 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1a10f0 con 0x7fa59c1011c0 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a4600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1a12b0 con 0x7fa59c1023b0 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a1200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c1a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a1200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47478/0 (socket says 172.21.15.2:47478) 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a1200640 1 -- 172.21.15.2:0/3620862828 learned_addr learned my addr 172.21.15.2:0/3620862828 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.232+0000 7fa5a1c00640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 0x7fa59c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 -- 172.21.15.2:0/3620862828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 msgr2=0x7fa59c199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:30.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 0x7fa59c199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 -- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 msgr2=0x7fa59c1a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c1a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 -- 172.21.15.2:0/3620862828 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa588009560 con 0x7fa59c1023b0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a2600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c1a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a1200640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa588006010 tx=0x7fa588005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5880043d0 con 0x7fa59c1023b0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5880052d0 con 0x7fa59c1023b0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588003cc0 con 0x7fa59c1023b0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.233+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c1a1530 con 0x7fa59c1023b0 2024-03-11T18:33:30.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.234+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c1a19f0 con 0x7fa59c1023b0 2024-03-11T18:33:30.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.235+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c10cbd0 con 0x7fa59c1023b0 2024-03-11T18:33:30.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.236+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7fa588005440 con 0x7fa59c1023b0 2024-03-11T18:33:30.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.238+0000 7fa59a000640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 0x7fa5800a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:30.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.238+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fa5880ef9b0 con 0x7fa59c1023b0 2024-03-11T18:33:30.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.238+0000 7fa5a1c00640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 0x7fa5800a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:30.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.239+0000 7fa5a1c00640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 0x7fa5800a3f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa59c102210 tx=0x7fa594009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:30.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.242+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa5880b87a0 con 0x7fa59c1023b0 2024-03-11T18:33:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:30 smithi002 ceph-mon[31328]: pgmap v415: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:30.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:30 smithi002 ceph-mon[31328]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:30 smithi165 ceph-mon[36634]: pgmap v415: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:30 smithi165 ceph-mon[36634]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:30 smithi023 ceph-mon[36406]: pgmap v415: 129 pgs: 129 active+clean; 643 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-11T18:33:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:30 smithi023 ceph-mon[36406]: from='mgr.14152 172.21.15.2:0/2641173413' entity='mgr.x' 2024-03-11T18:33:30.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:30.488+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fa59c10cd60 con 0x7fa59c1023b0 2024-03-11T18:33:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.049+0000 7fa59a000640 1 -- 172.21.15.2:0/3620862828 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa5880bbe60 con 0x7fa59c1023b0 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 msgr2=0x7fa5800a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 0x7fa5800a3f60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa59c102210 tx=0x7fa594009500 comp rx=0 tx=0).stop 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 msgr2=0x7fa59c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa588006010 tx=0x7fa588005e80 comp rx=0 tx=0).stop 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 shutdown_connections 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fa5800a1aa0 0x7fa5800a3f60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa59c102d70 0x7fa59c1a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa59c1023b0 0x7fa59c1999d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 --2- 172.21.15.2:0/3620862828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa59c1011c0 0x7fa59c199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.053+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 >> 172.21.15.2:0/3620862828 conn(0x7fa59c0fc750 msgr2=0x7fa59c0733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.054+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 shutdown_connections 2024-03-11T18:33:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.054+0000 7fa5a4600640 1 -- 172.21.15.2:0/3620862828 wait complete. 2024-03-11T18:33:31.082 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 -- 172.21.15.2:0/1064147570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc103200 msgr2=0x7f10fc103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 --2- 172.21.15.2:0/1064147570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc103200 0x7f10fc103620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f10ec009900 tx=0x7f10ec02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 -- 172.21.15.2:0/1064147570 shutdown_connections 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 --2- 172.21.15.2:0/1064147570 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc10b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 --2- 172.21.15.2:0/1064147570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc1043f0 0x7f10fc104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 --2- 172.21.15.2:0/1064147570 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc103200 0x7f10fc103620 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 -- 172.21.15.2:0/1064147570 >> 172.21.15.2:0/1064147570 conn(0x7f10fc0fe790 msgr2=0x7f10fc100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:31.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 -- 172.21.15.2:0/1064147570 shutdown_connections 2024-03-11T18:33:31.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.312+0000 7f1103000640 1 -- 172.21.15.2:0/1064147570 wait complete. 2024-03-11T18:33:31.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.313+0000 7f1103000640 1 Processor -- start 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 -- start start 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc1043f0 0x7f10fc199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1a0db0 con 0x7f10fc104db0 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1a0f20 con 0x7f10fc103200 2024-03-11T18:33:31.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1103000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc1a10e0 con 0x7f10fc1043f0 2024-03-11T18:33:31.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1101c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1102600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.314+0000 7f1101c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34510/0 (socket says 172.21.15.2:34510) 2024-03-11T18:33:31.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 -- 172.21.15.2:0/2488572441 learned_addr learned my addr 172.21.15.2:0/2488572441 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101200640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc1043f0 0x7f10fc199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 -- 172.21.15.2:0/2488572441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc1043f0 msgr2=0x7f10fc199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc1043f0 0x7f10fc199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 -- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 msgr2=0x7f10fc1a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc1a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 -- 172.21.15.2:0/2488572441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ec009560 con 0x7f10fc103200 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1102600640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.315+0000 7f1101c00640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10ec000c00 tx=0x7f10ec005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.316+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec0043d0 con 0x7f10fc103200 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.316+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc1a1360 con 0x7f10fc103200 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.316+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10ec0052d0 con 0x7f10fc103200 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.316+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ec003d30 con 0x7f10fc103200 2024-03-11T18:33:31.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.316+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc1a18c0 con 0x7f10fc103200 2024-03-11T18:33:31.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.317+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc10a520 con 0x7f10fc103200 2024-03-11T18:33:31.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.319+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7f10ec005440 con 0x7f10fc103200 2024-03-11T18:33:31.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.320+0000 7f10fa000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 0x7f10e00a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.320+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f10ec0f00c0 con 0x7f10fc103200 2024-03-11T18:33:31.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.321+0000 7f1101200640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 0x7f10e00a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.321+0000 7f1101200640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 0x7f10e00a3f50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f10fc19a7d0 tx=0x7f10e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:31.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.323+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f10ec0b8eb0 con 0x7f10fc103200 2024-03-11T18:33:31.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:31 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3620862828' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:31 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:31 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3620862828' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:31 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3620862828' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.526 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-11T18:33:31.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.554+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f10fc02cee0 con 0x7f10e00a1a90 2024-03-11T18:33:31.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.587+0000 7f10fa000640 1 -- 172.21.15.2:0/2488572441 <== mgr.14152 v2:172.21.15.2:6800/1446641624 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f10fc02cee0 con 0x7f10e00a1a90 2024-03-11T18:33:31.589 INFO:teuthology.orchestra.run.smithi002.stdout:{} 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 msgr2=0x7f10e00a3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 0x7f10e00a3f50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f10fc19a7d0 tx=0x7f10e8005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 msgr2=0x7f10fc1992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10ec000c00 tx=0x7f10ec005e80 comp rx=0 tx=0).stop 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 shutdown_connections 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f10e00a1a90 0x7f10e00a3f50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10fc104db0 0x7f10fc1a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10fc1043f0 0x7f10fc199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 --2- 172.21.15.2:0/2488572441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10fc103200 0x7f10fc1992c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 >> 172.21.15.2:0/2488572441 conn(0x7f10fc0fe790 msgr2=0x7f10fc1076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 shutdown_connections 2024-03-11T18:33:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.589+0000 7f1103000640 1 -- 172.21.15.2:0/2488572441 wait complete. 2024-03-11T18:33:31.609 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-11T18:33:31.624 INFO:teuthology.run_tasks:Running task exec... 2024-03-11T18:33:31.632 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T18:33:31.633 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:33:31.633 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 -- 172.21.15.2:0/1867948988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102d80 msgr2=0x7ff4bc109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 --2- 172.21.15.2:0/1867948988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102d80 0x7ff4bc109610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009ca0 tx=0x7ff4b402c0f0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 -- 172.21.15.2:0/1867948988 shutdown_connections 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 --2- 172.21.15.2:0/1867948988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc102d80 0x7ff4bc109610 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 --2- 172.21.15.2:0/1867948988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 --2- 172.21.15.2:0/1867948988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 -- 172.21.15.2:0/1867948988 >> 172.21.15.2:0/1867948988 conn(0x7ff4bc0fc740 msgr2=0x7ff4bc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 -- 172.21.15.2:0/1867948988 shutdown_connections 2024-03-11T18:33:31.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.841+0000 7ff4c2800640 1 -- 172.21.15.2:0/1867948988 wait complete. 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.842+0000 7ff4c2800640 1 Processor -- start 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.842+0000 7ff4c2800640 1 -- start start 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102d80 0x7ff4bc19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19ece0 con 0x7ff4bc102d80 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19ee50 con 0x7ff4bc1011d0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4c2800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19f010 con 0x7ff4bc1023c0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4bbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102d80 0x7ff4bc19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4bb400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47556/0 (socket says 172.21.15.2:47556) 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 -- 172.21.15.2:0/785062112 learned_addr learned my addr 172.21.15.2:0/785062112 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 -- 172.21.15.2:0/785062112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 msgr2=0x7ff4bc1971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 -- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102d80 msgr2=0x7ff4bc19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102d80 0x7ff4bc19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4b3e00640 1 -- 172.21.15.2:0/785062112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4009900 con 0x7ff4bc1023c0 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.843+0000 7ff4bb400640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:31.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4b3e00640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400e9f0 tx=0x7ff4a400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:31.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4009720 con 0x7ff4bc1023c0 2024-03-11T18:33:31.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc19f2f0 con 0x7ff4bc1023c0 2024-03-11T18:33:31.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4a400cd80 con 0x7ff4bc1023c0 2024-03-11T18:33:31.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4010690 con 0x7ff4bc1023c0 2024-03-11T18:33:31.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.844+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc19f800 con 0x7ff4bc1023c0 2024-03-11T18:33:31.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.845+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc1084f0 con 0x7ff4bc1023c0 2024-03-11T18:33:31.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.847+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 18) v1 ==== 136278+0+0 (secure 0 0 0) 0x7ff4a40046c0 con 0x7ff4bc1023c0 2024-03-11T18:33:31.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.849+0000 7ff4b8c00640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 0x7ff4a00a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:31.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.849+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7ff4a4014070 con 0x7ff4bc1023c0 2024-03-11T18:33:31.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.849+0000 7ff4bb400640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 0x7ff4a00a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:31.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.850+0000 7ff4bb400640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 0x7ff4a00a3ee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc102220 tx=0x7ff4ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:31.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:31.853+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff4a4096980 con 0x7ff4bc1023c0 2024-03-11T18:33:32.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:31.566+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:32.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:31.566+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:32.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:31.566+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:32.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:31.566+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:32.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:31.566+0000 7f2283c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:32.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:32.106+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff4bc02cdd0 con 0x7ff4bc1023c0 2024-03-11T18:33:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:32 smithi165 ceph-mon[36634]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:33:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:32 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T18:33:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:32 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:33:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:32 smithi023 ceph-mon[36406]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:33:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:32 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T18:33:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:32 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:33:32.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:32 smithi002 ceph-mon[31328]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 6 op/s 2024-03-11T18:33:32.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:32 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-11T18:33:32.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:32 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:33:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.079+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mgrmap(e 19) v1 ==== 136346+0+0 (secure 0 0 0) 0x7ff4a4096d50 con 0x7ff4bc1023c0 2024-03-11T18:33:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.080+0000 7ff4b8c00640 1 -- 172.21.15.2:0/785062112 <== mon.1 v2:172.21.15.165:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff4a404d5d0 con 0x7ff4bc1023c0 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 msgr2=0x7ff4a00a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 0x7ff4a00a3ee0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc102220 tx=0x7ff4ac005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 msgr2=0x7ff4bc197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400e9f0 tx=0x7ff4a400edf0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 shutdown_connections 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7ff4a00a1a20 0x7ff4a00a3ee0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4bc102d80 0x7ff4bc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1023c0 0x7ff4bc197730 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 --2- 172.21.15.2:0/785062112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4bc1011d0 0x7ff4bc1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.086+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 >> 172.21.15.2:0/785062112 conn(0x7ff4bc0fc740 msgr2=0x7ff4bc1056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:33.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.087+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 shutdown_connections 2024-03-11T18:33:33.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.087+0000 7ff4c2800640 1 -- 172.21.15.2:0/785062112 wait complete. 2024-03-11T18:33:33.108 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-11T18:33:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 -- 172.21.15.2:0/1190744688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 msgr2=0x7fc8e410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 --2- 172.21.15.2:0/1190744688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e410b820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009ca0 tx=0x7fc8dc02c0f0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 -- 172.21.15.2:0/1190744688 shutdown_connections 2024-03-11T18:33:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 --2- 172.21.15.2:0/1190744688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e410b820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 --2- 172.21.15.2:0/1190744688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41045d0 0x7fc8e4104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 --2- 172.21.15.2:0/1190744688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41033e0 0x7fc8e4103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.310+0000 7fc8ea000640 1 -- 172.21.15.2:0/1190744688 >> 172.21.15.2:0/1190744688 conn(0x7fc8e40fe970 msgr2=0x7fc8e4100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:33.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.311+0000 7fc8ea000640 1 -- 172.21.15.2:0/1190744688 shutdown_connections 2024-03-11T18:33:33.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.311+0000 7fc8ea000640 1 -- 172.21.15.2:0/1190744688 wait complete. 2024-03-11T18:33:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 Processor -- start 2024-03-11T18:33:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 -- start start 2024-03-11T18:33:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 0x7fc8e4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a0ee0 con 0x7fc8e41045d0 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a1050 con 0x7fc8e41033e0 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8ea000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e41a1210 con 0x7fc8e4104f90 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8e3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8e3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34574/0 (socket says 172.21.15.2:34574) 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8e3e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8e3400640 1 -- 172.21.15.2:0/2398854506 learned_addr learned my addr 172.21.15.2:0/2398854506 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.312+0000 7fc8dbe00640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 0x7fc8e4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 msgr2=0x7fc8e41a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e41a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 msgr2=0x7fc8e4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 0x7fc8e4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 -- 172.21.15.2:0/2398854506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc009490 con 0x7fc8e41033e0 2024-03-11T18:33:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8dbe00640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 0x7fc8e4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e3400640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400e9c0 tx=0x7fc8d400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4009700 con 0x7fc8e41033e0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.314+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc009900 con 0x7fc8e41033e0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.313+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8d400cd80 con 0x7fc8e41033e0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.314+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4010430 con 0x7fc8e41033e0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.314+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e41a1870 con 0x7fc8e41033e0 2024-03-11T18:33:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.315+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e410a700 con 0x7fc8e41033e0 2024-03-11T18:33:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.319+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 136346+0+0 (secure 0 0 0) 0x7fc8d4003680 con 0x7fc8e41033e0 2024-03-11T18:33:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.320+0000 7fc8e0c00640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 0x7fc8c80a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.320+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fc8d4014070 con 0x7fc8e41033e0 2024-03-11T18:33:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.320+0000 7fc8dbe00640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 msgr2=0x7fc8c80a3fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.320+0000 7fc8dbe00640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 0x7fc8c80a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:33:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.321+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc8d4096b20 con 0x7fc8e41033e0 2024-03-11T18:33:33.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-mon[36634]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:33.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/785062112' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: ignoring --setuser ceph since I am not root 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: ignoring --setgroup ceph since I am not root 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-mgr[38507]: -- 172.21.15.165:0/762308577 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb053421a0 con 0x55fb05318800 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-mon[36406]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/785062112' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.407 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: ignoring --setuser ceph since I am not root 2024-03-11T18:33:33.407 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: ignoring --setgroup ceph since I am not root 2024-03-11T18:33:33.407 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-mgr[37795]: -- 172.21.15.23:0/1278643431 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5599ec6f21a0 con 0x5599ec6c8c00 2024-03-11T18:33:33.502 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setuser ceph since I am not root 2024-03-11T18:33:33.503 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: ignoring --setgroup ceph since I am not root 2024-03-11T18:33:33.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-mon[31328]: from='client.25391 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:33.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/785062112' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-11T18:33:33.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.521+0000 7fc8dbe00640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 msgr2=0x7fc8c80a3fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:33.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.521+0000 7fc8dbe00640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 0x7fc8c80a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:33:33.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.522+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc8e410edf0 con 0x7fc8e41033e0 2024-03-11T18:33:33.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.533+0000 7fc8e0c00640 1 -- 172.21.15.2:0/2398854506 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc8d409a1e0 con 0x7fc8e41033e0 2024-03-11T18:33:33.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 msgr2=0x7fc8c80a3fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 0x7fc8c80a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 msgr2=0x7fc8e41993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400e9c0 tx=0x7fc8d400ee80 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 shutdown_connections 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fc8c80a1b00 0x7fc8c80a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8e4104f90 0x7fc8e41a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e41045d0 0x7fc8e4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.535+0000 7fc8ea000640 1 --2- 172.21.15.2:0/2398854506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e41033e0 0x7fc8e41993f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.536+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 >> 172.21.15.2:0/2398854506 conn(0x7fc8e40fe970 msgr2=0x7fc8e41078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.536+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 shutdown_connections 2024-03-11T18:33:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.536+0000 7fc8ea000640 1 -- 172.21.15.2:0/2398854506 wait complete. 2024-03-11T18:33:33.556 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 -- 172.21.15.2:0/2784872232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca501089f0 msgr2=0x7fca50108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 --2- 172.21.15.2:0/2784872232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca501089f0 0x7fca50108e70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009900 tx=0x7fca3c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 -- 172.21.15.2:0/2784872232 shutdown_connections 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 --2- 172.21.15.2:0/2784872232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 0x7fca5010fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 --2- 172.21.15.2:0/2784872232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca501089f0 0x7fca50108e70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 --2- 172.21.15.2:0/2784872232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca50107800 0x7fca50107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 -- 172.21.15.2:0/2784872232 >> 172.21.15.2:0/2784872232 conn(0x7fca500767d0 msgr2=0x7fca50078bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:33.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 -- 172.21.15.2:0/2784872232 shutdown_connections 2024-03-11T18:33:33.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.750+0000 7fca56400640 1 -- 172.21.15.2:0/2784872232 wait complete. 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.751+0000 7fca56400640 1 Processor -- start 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.751+0000 7fca56400640 1 -- start start 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.751+0000 7fca56400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.751+0000 7fca56400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca501089f0 0x7fca5019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca56400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 0x7fca501a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca56400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501a53a0 con 0x7fca501093b0 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca56400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501a5510 con 0x7fca50107800 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca56400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca501a56d0 con 0x7fca501089f0 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34600/0 (socket says 172.21.15.2:34600) 2024-03-11T18:33:33.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 -- 172.21.15.2:0/1810595150 learned_addr learned my addr 172.21.15.2:0/1810595150 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4ea00640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca501089f0 0x7fca5019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4fe00640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 0x7fca501a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca501089f0 msgr2=0x7fca5019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca501089f0 0x7fca5019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 msgr2=0x7fca501a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 0x7fca501a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 -- 172.21.15.2:0/1810595150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009560 con 0x7fca50107800 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.752+0000 7fca4f400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fca4000e9c0 tx=0x7fca4000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.753+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca400096e0 con 0x7fca50107800 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.753+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca4000cd80 con 0x7fca50107800 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.753+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40010430 con 0x7fca50107800 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.753+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca501a59b0 con 0x7fca50107800 2024-03-11T18:33:33.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.753+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca501a5e40 con 0x7fca50107800 2024-03-11T18:33:33.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.754+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5010eb40 con 0x7fca50107800 2024-03-11T18:33:33.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.756+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 136346+0+0 (secure 0 0 0) 0x7fca40003680 con 0x7fca50107800 2024-03-11T18:33:33.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.757+0000 7fca47e00640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 0x7fca340a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:33.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.757+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7fca40014070 con 0x7fca50107800 2024-03-11T18:33:33.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.757+0000 7fca4ea00640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 msgr2=0x7fca340a3fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.757+0000 7fca4ea00640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 0x7fca340a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:33:33.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.761+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fca40096a30 con 0x7fca50107800 2024-03-11T18:33:33.905 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:33.642+0000 7fbab1738200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:33:33.905 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:33.736+0000 7fbab1738200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:33:33.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.958+0000 7fca4ea00640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 msgr2=0x7fca340a3fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:33.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.958+0000 7fca4ea00640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 0x7fca340a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:33:33.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.962+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fca50113200 con 0x7fca50107800 2024-03-11T18:33:33.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.969+0000 7fca47e00640 1 -- 172.21.15.2:0/1810595150 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) v1 ==== 133+0+0 (secure 0 0 0) 0x7fca4009a0f0 con 0x7fca50107800 2024-03-11T18:33:33.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.971+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 msgr2=0x7fca340a3fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:33:33.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.971+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 0x7fca340a3fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.971+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 msgr2=0x7fca5019d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.971+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fca4000e9c0 tx=0x7fca4000ee80 comp rx=0 tx=0).stop 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 shutdown_connections 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7fca340a1b00 0x7fca340a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca501093b0 0x7fca501a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca501089f0 0x7fca5019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 --2- 172.21.15.2:0/1810595150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca50107800 0x7fca5019d8b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 >> 172.21.15.2:0/1810595150 conn(0x7fca500767d0 msgr2=0x7fca5010bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 shutdown_connections 2024-03-11T18:33:33.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:33.972+0000 7fca56400640 1 -- 172.21.15.2:0/1810595150 wait complete. 2024-03-11T18:33:33.992 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-11T18:33:34.071 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:33.660+0000 7f4d7909e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:33:34.071 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:33.757+0000 7f4d7909e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:33:34.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 -- 172.21.15.2:0/1542109725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 msgr2=0x7f3560103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:34.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 --2- 172.21.15.2:0/1542109725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f3560103e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3550009900 tx=0x7f355002bc40 comp rx=0 tx=0).stop 2024-03-11T18:33:34.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 -- 172.21.15.2:0/1542109725 shutdown_connections 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 --2- 172.21.15.2:0/1542109725 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3560106cd0 0x7f3560109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 --2- 172.21.15.2:0/1542109725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35601043a0 0x7f3560106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 --2- 172.21.15.2:0/1542109725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f3560103e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 -- 172.21.15.2:0/1542109725 >> 172.21.15.2:0/1542109725 conn(0x7f35600fb440 msgr2=0x7f35600fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.189+0000 7f355f200640 1 -- 172.21.15.2:0/1542109725 shutdown_connections 2024-03-11T18:33:34.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.190+0000 7f355f200640 1 -- 172.21.15.2:0/1542109725 wait complete. 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.190+0000 7f355f200640 1 Processor -- start 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 -- start start 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f35601992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 0x7f35601997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3560106cd0 0x7f35601a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35601a0d90 con 0x7f3560106cd0 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35601a0f00 con 0x7f3560101a70 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35601a10c0 con 0x7f35601043a0 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 0x7f35601997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:34.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f35601992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f35601992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34620/0 (socket says 172.21.15.2:34620) 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 learned_addr learned my addr 172.21.15.2:0/1805373346 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355d400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 0x7f35601997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47626/0 (socket says 172.21.15.2:47626) 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.191+0000 7f355e800640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3560106cd0 0x7f35601a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 msgr2=0x7f35601992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3560101a70 0x7f35601992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3560106cd0 msgr2=0x7f35601a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3560106cd0 0x7f35601a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558009840 con 0x7f35601043a0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355d400640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 0x7f35601997e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f354c00e9c0 tx=0x7f354c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c00ccc0 con 0x7f35601043a0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550009560 con 0x7f35601043a0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f354c00ce20 con 0x7f35601043a0 2024-03-11T18:33:34.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c018880 con 0x7f35601043a0 2024-03-11T18:33:34.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.192+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35601a1770 con 0x7f35601043a0 2024-03-11T18:33:34.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.194+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3560108820 con 0x7f35601043a0 2024-03-11T18:33:34.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.196+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 19) v1 ==== 136346+0+0 (secure 0 0 0) 0x7f354c0189e0 con 0x7f35601043a0 2024-03-11T18:33:34.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.197+0000 7f3556a00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:34.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.198+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9187+0+0 (secure 0 0 0) 0x7f354c014070 con 0x7f35601043a0 2024-03-11T18:33:34.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.198+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:34.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.198+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T18:33:34.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.206+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f354c052d20 con 0x7f35601043a0 2024-03-11T18:33:34.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:33.926+0000 7faf70913200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T18:33:34.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:34.059+0000 7faf70913200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T18:33:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T18:33:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-mon[31328]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T18:33:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.386 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T18:33:34.386 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-mon[36634]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T18:33:34.386 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.386 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.399+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.399+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T18:33:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-11T18:33:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-mon[36406]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-03-11T18:33:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' 2024-03-11T18:33:34.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.428+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f35600615c0 con 0x7f35440a1af0 2024-03-11T18:33:34.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:34.384+0000 7fbab1738200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:33:34.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:34.477+0000 7fbab1738200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:33:34.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:34.620+0000 7fbab1738200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:33:34.766 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:34.422+0000 7f4d7909e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:33:34.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:34.519+0000 7f4d7909e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:33:34.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:34.664+0000 7f4d7909e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:33:34.767 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:34.764+0000 7f4d7909e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:33:34.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.800+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:34.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:34.800+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T18:33:35.065 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:34.715+0000 7fbab1738200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:33:35.065 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:34.799+0000 7fbab1738200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:33:35.065 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:35.063+0000 7fbab1738200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:33:35.125 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:34.851+0000 7f4d7909e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:33:35.193 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:34.881+0000 7faf70913200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T18:33:35.193 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.008+0000 7faf70913200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T18:33:35.193 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.189+0000 7faf70913200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T18:33:35.354 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:35.172+0000 7fbab1738200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:35.354 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:35.258+0000 7fbab1738200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:33:35.354 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:35.352+0000 7fbab1738200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:33:35.405 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:35.122+0000 7f4d7909e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:33:35.405 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:35.235+0000 7f4d7909e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:35.405 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:35.324+0000 7f4d7909e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:33:35.502 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.320+0000 7faf70913200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T18:33:35.503 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.434+0000 7faf70913200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T18:33:35.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:35.601+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:35.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:35.601+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T18:33:35.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:35.528+0000 7fbab1738200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:33:35.905 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:35.423+0000 7f4d7909e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:33:35.905 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:35.605+0000 7f4d7909e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:33:36.047 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.779+0000 7faf70913200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T18:33:36.047 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:35.926+0000 7faf70913200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:36.403 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:36.043+0000 7faf70913200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T18:33:36.403 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:36.172+0000 7faf70913200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T18:33:36.403 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:36.399+0000 7faf70913200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T18:33:36.905 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:36.486+0000 7fbab1738200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:33:36.905 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:36.616+0000 7fbab1738200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:33:36.906 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:36.619+0000 7f4d7909e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:33:36.906 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:36.755+0000 7f4d7909e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:33:37.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:37.203+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:37.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:37.203+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T18:33:37.236 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:36.951+0000 7fbab1738200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:33:37.237 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:37.041+0000 7fbab1738200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:33:37.397 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:37.099+0000 7f4d7909e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:33:37.397 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:37.191+0000 7f4d7909e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:33:37.397 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:37.394+0000 7f4d7909e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:37.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:37.234+0000 7fbab1738200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:37.655 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:37.419+0000 7fbab1738200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:33:37.656 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:37.586+0000 7f4d7909e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:33:37.916 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:37 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:37.732+0000 7faf70913200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T18:33:37.916 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:37 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:37.913+0000 7faf70913200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T18:33:37.969 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:37.705+0000 7fbab1738200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:38.151 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:37.880+0000 7f4d7909e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:38.405 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:37.966+0000 7fbab1738200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:38.405 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:38.203+0000 7fbab1738200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:33:38.405 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:38.149+0000 7f4d7909e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:38.406 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:38.393+0000 7f4d7909e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:33:38.741 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:38.343+0000 7faf70913200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T18:33:38.741 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:38.468+0000 7faf70913200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T18:33:38.997 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:38.738+0000 7faf70913200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:38.997 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:38 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:38.994+0000 7faf70913200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T18:33:39.405 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:39.000+0000 7fbab1738200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:33:39.655 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:39.226+0000 7f4d7909e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:33:39.701 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:39.389+0000 7faf70913200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T18:33:39.893 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:39.579+0000 7fbab1738200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:33:39.893 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:39.677+0000 7fbab1738200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:33:39.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:39.975+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mgrmap(e 20) v1 ==== 136346+0+0 (secure 0 0 0) 0x7f354c096e00 con 0x7f35601043a0 2024-03-11T18:33:39.985 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:39.723+0000 7faf70913200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T18:33:40.154 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:39.820+0000 7f4d7909e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:33:40.154 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:39.921+0000 7f4d7909e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:33:40.154 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y[37791]: 2024-03-11T18:33:40.151+0000 7f4d7909e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: Standby manager daemon z restarted 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: Standby manager daemon z started 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:39 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:40.155 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z[38503]: 2024-03-11T18:33:39.892+0000 7fbab1738200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: Standby manager daemon z restarted 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: Standby manager daemon z started 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:33:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:39 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:40.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:40.048+0000 7faf70913200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T18:33:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: Standby manager daemon z restarted 2024-03-11T18:33:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: Standby manager daemon z started 2024-03-11T18:33:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T18:33:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T18:33:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:39 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.165:0/54954475' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:40.406+0000 7f355de00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.2:6800/1446641624 2024-03-11T18:33:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:40.406+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-11T18:33:41.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:41.011+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 8 ==== mgrmap(e 21) v1 ==== 136346+0+0 (secure 0 0 0) 0x7f354c0cb580 con 0x7f35601043a0 2024-03-11T18:33:41.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:41 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:41.135+0000 7faf70913200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: Standby manager daemon y restarted 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: Standby manager daemon y started 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:33:41.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:40 smithi002 ceph-mon[31328]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T18:33:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: Standby manager daemon y restarted 2024-03-11T18:33:41.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: Standby manager daemon y started 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:40 smithi023 ceph-mon[36406]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: Standby manager daemon y restarted 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: Standby manager daemon y started 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-11T18:33:41.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T18:33:41.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-11T18:33:41.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:40 smithi165 ceph-mon[36634]: from='mgr.? 172.21.15.23:0/3979007687' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T18:33:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-mon[31328]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-11T18:33:42.252 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:41 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:41.872+0000 7faf70913200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T18:33:42.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.012+0000 7faf70913200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T18:33:42.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:42.336+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 9 ==== mgrmap(e 22) v1 ==== 135964+0+0 (secure 0 0 0) 0x7f354c0cae80 con 0x7f35601043a0 2024-03-11T18:33:42.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:42.338+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 msgr2=0x7f35440a3fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:33:42.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:42.338+0000 7f3556a00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1446641624,v1:172.21.15.2:6801/1446641624] conn(0x7f35440a1af0 0x7f35440a3fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:42.340 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:42 smithi165 ceph-mon[36634]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-11T18:33:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:42 smithi023 ceph-mon[36406]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-11T18:33:42.628 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.310+0000 7faf70913200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T18:33:43.002 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.625+0000 7faec6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.626+0000 7faec6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.626+0000 7faec6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.626+0000 7faec6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.626+0000 7faec6e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.627+0000 7faec3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.627+0000 7faec3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.627+0000 7faec3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.627+0000 7faec3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:33:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:33:42.627+0000 7faec3200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: Active manager daemon x restarted 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: Activating manager daemon x 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: mgrmap e22: x(active, starting, since 0.0231401s), standbys: y, z 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:33:43.302 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:33:43.303 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:33:43.304 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: Manager daemon x is now available 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.305 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:43 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.342+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mon.1 v2:172.21.15.165:3300/0 10 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f354c0976d0 con 0x7f35601043a0 2024-03-11T18:33:43.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.343+0000 7f3556a00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f354404c4a0 0x7f354404c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:43.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.343+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f35600615c0 con 0x7f354404c4a0 2024-03-11T18:33:43.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.353+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f354404c4a0 0x7f354404c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:43.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.359+0000 7f355de00640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f354404c4a0 0x7f354404c880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3550006010 tx=0x7f355002c7f0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: Active manager daemon x restarted 2024-03-11T18:33:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: Activating manager daemon x 2024-03-11T18:33:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T18:33:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: mgrmap e22: x(active, starting, since 0.0231401s), standbys: y, z 2024-03-11T18:33:43.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:33:43.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:33:43.408 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:33:43.409 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: Manager daemon x is now available 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.410 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:43 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.447+0000 7f3556a00640 1 -- 172.21.15.2:0/1805373346 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35600615c0 con 0x7f354404c4a0 2024-03-11T18:33:43.449 INFO:teuthology.orchestra.run.smithi002.stdout:Schedule set for path / 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f354404c4a0 msgr2=0x7f354404c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f354404c4a0 0x7f354404c880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3550006010 tx=0x7f355002c7f0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 msgr2=0x7f35601997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 --2- 172.21.15.2:0/1805373346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35601043a0 0x7f35601997e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f354c00e9c0 tx=0x7f354c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355e800640 1 -- 172.21.15.2:0/1805373346 reap_dead start 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 shutdown_connections 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 >> 172.21.15.2:0/1805373346 conn(0x7f35600fb440 msgr2=0x7f35601010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 shutdown_connections 2024-03-11T18:33:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.451+0000 7f355f200640 1 -- 172.21.15.2:0/1805373346 wait complete. 2024-03-11T18:33:43.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: Active manager daemon x restarted 2024-03-11T18:33:43.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: Activating manager daemon x 2024-03-11T18:33:43.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: osdmap e96: 12 total, 12 up, 12 in 2024-03-11T18:33:43.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: mgrmap e22: x(active, starting, since 0.0231401s), standbys: y, z 2024-03-11T18:33:43.469 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:33:43.470 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-11T18:33:43.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: Manager daemon x is now available 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-11T18:33:43.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.473 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:43 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-11T18:33:43.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-11T18:33:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 -- 172.21.15.2:0/3033040885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 msgr2=0x7f8a04109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 --2- 172.21.15.2:0/3033040885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a04109610 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009900 tx=0x7f89fc02bc30 comp rx=0 tx=0).stop 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 -- 172.21.15.2:0/3033040885 shutdown_connections 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 --2- 172.21.15.2:0/3033040885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a04109610 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 --2- 172.21.15.2:0/3033040885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041023c0 0x7f8a04102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 --2- 172.21.15.2:0/3033040885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041011d0 0x7f8a041015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 -- 172.21.15.2:0/3033040885 >> 172.21.15.2:0/3033040885 conn(0x7f8a040fc740 msgr2=0x7f8a040feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 -- 172.21.15.2:0/3033040885 shutdown_connections 2024-03-11T18:33:43.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.661+0000 7f8a09600640 1 -- 172.21.15.2:0/3033040885 wait complete. 2024-03-11T18:33:43.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.662+0000 7f8a09600640 1 Processor -- start 2024-03-11T18:33:43.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.662+0000 7f8a09600640 1 -- start start 2024-03-11T18:33:43.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.662+0000 7f8a09600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a09600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 0x7f8a04199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a09600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a09600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041a0f20 con 0x7f8a04102d80 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a09600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041a1090 con 0x7f8a041011d0 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a09600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a041a1250 con 0x7f8a041023c0 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a02a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f89fa000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 0x7f8a04199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56472/0 (socket says 172.21.15.2:56472) 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 -- 172.21.15.2:0/3976917389 learned_addr learned my addr 172.21.15.2:0/3976917389 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a02a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50092/0 (socket says 172.21.15.2:50092) 2024-03-11T18:33:43.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 -- 172.21.15.2:0/3976917389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 msgr2=0x7f8a04199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 0x7f8a04199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 -- 172.21.15.2:0/3976917389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 msgr2=0x7f8a04199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a03400640 1 -- 172.21.15.2:0/3976917389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc009560 con 0x7f8a04102d80 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f89fa000640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 0x7f8a04199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.663+0000 7f8a02a00640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f8a03400640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009230 tx=0x7f89fc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc004440 con 0x7f8a04102d80 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a041a14d0 con 0x7f8a04102d80 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89fc0052d0 con 0x7f8a04102d80 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89fc003cc0 con 0x7f8a04102d80 2024-03-11T18:33:43.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.664+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a041a1a30 con 0x7f8a04102d80 2024-03-11T18:33:43.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.666+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0410cbb0 con 0x7f8a04102d80 2024-03-11T18:33:43.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.667+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f89fc005440 con 0x7f8a04102d80 2024-03-11T18:33:43.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.669+0000 7f89fbe00640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 0x7f89e80a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:43.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.669+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f89fc0ef750 con 0x7f8a04102d80 2024-03-11T18:33:43.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.669+0000 7f8a02a00640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 0x7f89e80a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:43.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.670+0000 7f8a02a00640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 0x7f89e80a3cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a04102220 tx=0x7f89f4009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:43.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.673+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f89fc0b8750 con 0x7f8a04102d80 2024-03-11T18:33:43.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:43.921+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a04074ad0 con 0x7f89e80a1810 2024-03-11T18:33:44.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.055+0000 7f89fbe00640 1 -- 172.21.15.2:0/3976917389 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f8a04074ad0 con 0x7f89e80a1810 2024-03-11T18:33:44.057 INFO:teuthology.orchestra.run.smithi002.stdout:Retention added to path / 2024-03-11T18:33:44.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 msgr2=0x7f89e80a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 0x7f89e80a3cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a04102220 tx=0x7f89f4009500 comp rx=0 tx=0).stop 2024-03-11T18:33:44.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 msgr2=0x7f8a041a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f89fc009230 tx=0x7f89fc004040 comp rx=0 tx=0).stop 2024-03-11T18:33:44.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 shutdown_connections 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f89e80a1810 0x7f89e80a3cd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8a04102d80 0x7f8a041a09e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a041023c0 0x7f8a04199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 --2- 172.21.15.2:0/3976917389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a041011d0 0x7f8a04199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 >> 172.21.15.2:0/3976917389 conn(0x7f8a040fc740 msgr2=0x7f8a040733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.059+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 shutdown_connections 2024-03-11T18:33:44.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.060+0000 7f8a09600640 1 -- 172.21.15.2:0/3976917389 wait complete. 2024-03-11T18:33:44.082 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-11T18:33:44.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 -- 172.21.15.2:0/2786848163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec073be0 msgr2=0x7f32ec074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 --2- 172.21.15.2:0/2786848163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec073be0 0x7f32ec074060 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009900 tx=0x7f32d802bc50 comp rx=0 tx=0).stop 2024-03-11T18:33:44.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 -- 172.21.15.2:0/2786848163 shutdown_connections 2024-03-11T18:33:44.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 --2- 172.21.15.2:0/2786848163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 --2- 172.21.15.2:0/2786848163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec073be0 0x7f32ec074060 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 --2- 172.21.15.2:0/2786848163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec0751c0 0x7f32ec073610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 -- 172.21.15.2:0/2786848163 >> 172.21.15.2:0/2786848163 conn(0x7f32ec0fc590 msgr2=0x7f32ec0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 -- 172.21.15.2:0/2786848163 shutdown_connections 2024-03-11T18:33:44.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.274+0000 7f32f1a00640 1 -- 172.21.15.2:0/2786848163 wait complete. 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 Processor -- start 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 -- start start 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 0x7f32ec1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1a0d70 con 0x7f32ec073be0 2024-03-11T18:33:44.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1a0ee0 con 0x7f32ec108ec0 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.276+0000 7f32f1a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec1a10a0 con 0x7f32ec0751c0 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32eb400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32ebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 0x7f32ec1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50104/0 (socket says 172.21.15.2:50104) 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32eb400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51298/0 (socket says 172.21.15.2:51298) 2024-03-11T18:33:44.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 -- 172.21.15.2:0/1242576018 learned_addr learned my addr 172.21.15.2:0/1242576018 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:44.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 -- 172.21.15.2:0/1242576018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 msgr2=0x7f32ec199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 -- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 msgr2=0x7f32ec1992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 0x7f32ec1992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 -- 172.21.15.2:0/1242576018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32dc009490 con 0x7f32ec108ec0 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32ebe00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 0x7f32ec1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32eb400640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.277+0000 7f32f1000640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eb10 tx=0x7f32e400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.278+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4004b20 con 0x7f32ec108ec0 2024-03-11T18:33:44.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.278+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e4004c80 con 0x7f32ec108ec0 2024-03-11T18:33:44.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.278+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4010650 con 0x7f32ec108ec0 2024-03-11T18:33:44.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.278+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d8009560 con 0x7f32ec108ec0 2024-03-11T18:33:44.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.278+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec1a1680 con 0x7f32ec108ec0 2024-03-11T18:33:44.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.279+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec0679f0 con 0x7f32ec108ec0 2024-03-11T18:33:44.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.281+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 136091+0+0 (secure 0 0 0) 0x7f32e4005020 con 0x7f32ec108ec0 2024-03-11T18:33:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.283+0000 7f32e8c00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 0x7f32d00a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.283+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f32e4014070 con 0x7f32ec108ec0 2024-03-11T18:33:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.283+0000 7f32ebe00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 0x7f32d00a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.284+0000 7f32ebe00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 0x7f32d00a3dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32dc011630 tx=0x7f32dc005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.287+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f32e4096a30 con 0x7f32ec108ec0 2024-03-11T18:33:44.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.356+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 24) v1 ==== 136398+0+0 (secure 0 0 0) 0x7f32e4096e00 con 0x7f32ec108ec0 2024-03-11T18:33:44.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.523+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f32ec02ce60 con 0x7f32d00a1910 2024-03-11T18:33:44.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.540+0000 7f32e8c00640 1 -- 172.21.15.2:0/1242576018 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f32ec02ce60 con 0x7f32d00a1910 2024-03-11T18:33:44.542 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-11T00:00:00", "created": "2024-03-11T18:33:43", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 msgr2=0x7f32d00a3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 0x7f32d00a3dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f32dc011630 tx=0x7f32dc005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 msgr2=0x7f32ec1a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eb10 tx=0x7f32e400c2d0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 shutdown_connections 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32ec108ec0 0x7f32ec1a0830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32ec0751c0 0x7f32ec199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f32d00a1910 0x7f32d00a3dd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 --2- 172.21.15.2:0/1242576018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32ec073be0 0x7f32ec1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 >> 172.21.15.2:0/1242576018 conn(0x7f32ec0fc590 msgr2=0x7f32ec104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:44.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 shutdown_connections 2024-03-11T18:33:44.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.543+0000 7f32f1a00640 1 -- 172.21.15.2:0/1242576018 wait complete. 2024-03-11T18:33:44.565 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-11T18:33:44.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: mgrmap e23: x(active, since 1.03835s), standbys: y, z 2024-03-11T18:33:44.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: [11/Mar/2024:18:33:43] ENGINE Bus STARTING 2024-03-11T18:33:44.645 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: [11/Mar/2024:18:33:43] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T18:33:44.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: [11/Mar/2024:18:33:43] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T18:33:44.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: [11/Mar/2024:18:33:43] ENGINE Bus STARTED 2024-03-11T18:33:44.646 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:44 smithi002 ceph-mon[31328]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: mgrmap e23: x(active, since 1.03835s), standbys: y, z 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: [11/Mar/2024:18:33:43] ENGINE Bus STARTING 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: [11/Mar/2024:18:33:43] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: [11/Mar/2024:18:33:43] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: [11/Mar/2024:18:33:43] ENGINE Bus STARTED 2024-03-11T18:33:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:44 smithi165 ceph-mon[36634]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: mgrmap e23: x(active, since 1.03835s), standbys: y, z 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: [11/Mar/2024:18:33:43] ENGINE Bus STARTING 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: [11/Mar/2024:18:33:43] ENGINE Serving on http://172.21.15.2:8765 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: [11/Mar/2024:18:33:43] ENGINE Serving on https://172.21.15.2:7150 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: [11/Mar/2024:18:33:43] ENGINE Bus STARTED 2024-03-11T18:33:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:44 smithi023 ceph-mon[36406]: from='client.15375 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 -- 172.21.15.2:0/1482587648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 msgr2=0x7f9820109610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 --2- 172.21.15.2:0/1482587648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f9820109610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9818009be0 tx=0x7f981802bef0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 -- 172.21.15.2:0/1482587648 shutdown_connections 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 --2- 172.21.15.2:0/1482587648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f9820109610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 --2- 172.21.15.2:0/1482587648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201023c0 0x7f9820102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 --2- 172.21.15.2:0/1482587648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201011d0 0x7f98201015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 -- 172.21.15.2:0/1482587648 >> 172.21.15.2:0/1482587648 conn(0x7f98200fc740 msgr2=0x7f98200feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 -- 172.21.15.2:0/1482587648 shutdown_connections 2024-03-11T18:33:44.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.777+0000 7f9825000640 1 -- 172.21.15.2:0/1482587648 wait complete. 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.778+0000 7f9825000640 1 Processor -- start 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.778+0000 7f9825000640 1 -- start start 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201011d0 0x7f9820199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 0x7f9820199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201a0f20 con 0x7f98201023c0 2024-03-11T18:33:44.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201a1090 con 0x7f9820102d80 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f9825000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98201a1250 con 0x7f98201011d0 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 0x7f9820199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50138/0 (socket says 172.21.15.2:50138) 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 -- 172.21.15.2:0/1966632416 learned_addr learned my addr 172.21.15.2:0/1966632416 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 0x7f9820199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56540/0 (socket says 172.21.15.2:56540) 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981de00640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201011d0 0x7f9820199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 -- 172.21.15.2:0/1966632416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201011d0 msgr2=0x7f9820199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201011d0 0x7f9820199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 -- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 msgr2=0x7f9820199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 0x7f9820199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:44.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.779+0000 7f981e800640 1 -- 172.21.15.2:0/1966632416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009560 con 0x7f9820102d80 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f981e800640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9818033420 tx=0x7f9818033cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818003b80 con 0x7f9820102d80 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9818003ce0 con 0x7f9820102d80 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180348c0 con 0x7f9820102d80 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9818009840 con 0x7f9820102d80 2024-03-11T18:33:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.780+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201a17a0 con 0x7f9820102d80 2024-03-11T18:33:44.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.782+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982010cc00 con 0x7f9820102d80 2024-03-11T18:33:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.783+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 24) v1 ==== 136398+0+0 (secure 0 0 0) 0x7f9818034a20 con 0x7f9820102d80 2024-03-11T18:33:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.784+0000 7f9816a00640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 0x7f98040a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.785+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f98180f0220 con 0x7f9820102d80 2024-03-11T18:33:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.785+0000 7f981de00640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 0x7f98040a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.786+0000 7f981de00640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 0x7f98040a4050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9810009530 tx=0x7f9810009260 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:44.789+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98180b8eb0 con 0x7f9820102d80 2024-03-11T18:33:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.022+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f98201a19c0 con 0x7f98040a1b90 2024-03-11T18:33:45.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.032+0000 7f9816a00640 1 -- 172.21.15.2:0/1966632416 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f98201a19c0 con 0x7f98040a1b90 2024-03-11T18:33:45.240 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1m 6m3h 2024-03-11T18:33:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 msgr2=0x7f98040a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 0x7f98040a4050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9810009530 tx=0x7f9810009260 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 msgr2=0x7f98201a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9818033420 tx=0x7f9818033cb0 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 shutdown_connections 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9820102d80 0x7f98201a09e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98040a1b90 0x7f98040a4050 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98201023c0 0x7f9820199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 --2- 172.21.15.2:0/1966632416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f98201011d0 0x7f9820199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 >> 172.21.15.2:0/1966632416 conn(0x7f98200fc740 msgr2=0x7f98200733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 shutdown_connections 2024-03-11T18:33:45.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:33:45.035+0000 7f9825000640 1 -- 172.21.15.2:0/1966632416 wait complete. 2024-03-11T18:33:45.243 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-11T18:33:45.331 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-11T18:33:45.825 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-11T18:33:46.493 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:33:46.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-11T18:33:46.495 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-11T18:33:46.495 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:46.495 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.495 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.495 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:45 smithi023 ceph-mon[36406]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.496 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:45 smithi165 ceph-mon[36634]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:46.497 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:45 smithi002 ceph-mon[31328]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:33:46.505 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T18:33:46.936 INFO:tasks.workunit:Pulling workunits from ref b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T18:33:46.936 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T18:33:46.936 INFO:tasks.workunit:timeout=3h 2024-03-11T18:33:46.936 INFO:tasks.workunit:cleanup=True 2024-03-11T18:33:46.937 DEBUG:teuthology.orchestra.run.smithi002:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:46.991 INFO:teuthology.orchestra.run.smithi002.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:46.991 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-03-11 18:31:03.456900288 +0000 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-03-11 18:31:03.456900288 +0000 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-03-11 18:33:29.646942829 +0000 2024-03-11T18:33:46.992 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-03-11 18:31:03.456900288 +0000 2024-03-11T18:33:46.992 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-11T18:33:46.992 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-11T18:33:47.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:33:47.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:46 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:33:47.103 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b6a4997be2b8bb74411268cc42753f29ae9a78ad 2024-03-11T18:33:47.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:33:47.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:33:47.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:46 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.159 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:47.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:33:47.160 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:46 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:33:47.190 INFO:tasks.workunit.client.0.smithi002.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T18:33:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:47 smithi165 ceph-mon[36634]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:47 smithi165 ceph-mon[36634]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-11T18:33:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:47 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:47 smithi023 ceph-mon[36406]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:47 smithi023 ceph-mon[36406]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-11T18:33:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:47 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:48.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:47 smithi002 ceph-mon[31328]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:33:48.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:47 smithi002 ceph-mon[31328]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-11T18:33:48.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:47 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:49.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:49.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T18:33:49.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:48 smithi165 ceph-mon[36634]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T18:33:49.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:48 smithi023 ceph-mon[36406]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T18:33:49.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:49.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: Updating smithi002:/etc/ceph/ceph.conf 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-11T18:33:49.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:48 smithi002 ceph-mon[31328]: Updating smithi165:/etc/ceph/ceph.conf 2024-03-11T18:33:50.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:49 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:49 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s rd, 205 B/s wr, 1 op/s 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: Updating smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:49 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:51.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:50 smithi165 ceph-mon[36634]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:51.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:50 smithi165 ceph-mon[36634]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:51.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:50 smithi023 ceph-mon[36406]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:51.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:50 smithi023 ceph-mon[36406]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:51.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:50 smithi002 ceph-mon[31328]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-03-11T18:33:51.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:50 smithi002 ceph-mon[31328]: Updating smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T18:33:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:52 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:52.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:52.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:52.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:52 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 76 op/s 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:33:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:52 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:33:54.308 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:54 smithi023 ceph-mon[36406]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-11T18:33:54.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:54 smithi165 ceph-mon[36634]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-11T18:33:54.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:54 smithi002 ceph-mon[31328]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-11T18:33:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:56 smithi165 ceph-mon[36634]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-11T18:33:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:56 smithi023 ceph-mon[36406]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-11T18:33:56.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:56 smithi002 ceph-mon[31328]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-11T18:33:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:33:58 smithi165 ceph-mon[36634]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-11T18:33:58.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:33:58 smithi023 ceph-mon[36406]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-11T18:33:58.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:33:58 smithi002 ceph-mon[31328]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-11T18:33:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 -- 172.21.15.2:0/971981399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81084b0 msgr2=0x7f9eb8110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/971981399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81084b0 0x7f9eb8110080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009cd0 tx=0x7f9eb002c0f0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 -- 172.21.15.2:0/971981399 shutdown_connections 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/971981399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81084b0 0x7f9eb8110080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/971981399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb8103d90 0x7f9eb8107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/971981399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb81033e0 0x7f9eb81037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.764+0000 7f9eb7e00640 1 -- 172.21.15.2:0/971981399 >> 172.21.15.2:0/971981399 conn(0x7f9eb80fec40 msgr2=0x7f9eb8101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.765+0000 7f9eb7e00640 1 -- 172.21.15.2:0/971981399 shutdown_connections 2024-03-11T18:33:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.765+0000 7f9eb7e00640 1 -- 172.21.15.2:0/971981399 wait complete. 2024-03-11T18:33:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 Processor -- start 2024-03-11T18:33:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 -- start start 2024-03-11T18:33:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81033e0 0x7f9eb819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb81084b0 0x7f9eb819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb819a150 con 0x7f9eb81033e0 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb819a2c0 con 0x7f9eb8199680 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.766+0000 7f9eb7e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb819a480 con 0x7f9eb81084b0 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81033e0 0x7f9eb819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38236/0 (socket says 172.21.15.2:38236) 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 -- 172.21.15.2:0/2869263555 learned_addr learned my addr 172.21.15.2:0/2869263555 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb6000640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb81084b0 0x7f9eb819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 -- 172.21.15.2:0/2869263555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb81084b0 msgr2=0x7f9eb819fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb81084b0 0x7f9eb819fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 -- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81033e0 msgr2=0x7f9eb819f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81033e0 0x7f9eb819f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 -- 172.21.15.2:0/2869263555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8009490 con 0x7f9eb8199680 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.767+0000 7f9eb7400640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0005e30 tx=0x7f9eb00325e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.768+0000 7f9eaf400640 1 -- 172.21.15.2:0/2869263555 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0034e20 con 0x7f9eb8199680 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.768+0000 7f9eaf400640 1 -- 172.21.15.2:0/2869263555 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eb0003e50 con 0x7f9eb8199680 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.768+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb0009900 con 0x7f9eb8199680 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.768+0000 7f9eaf400640 1 -- 172.21.15.2:0/2869263555 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0004750 con 0x7f9eb8199680 2024-03-11T18:33:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.768+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb81a6660 con 0x7f9eb8199680 2024-03-11T18:33:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.771+0000 7f9eaf400640 1 -- 172.21.15.2:0/2869263555 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 136454+0+0 (secure 0 0 0) 0x7f9eb000a040 con 0x7f9eb8199680 2024-03-11T18:33:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.771+0000 7f9ec076a740 1 Processor -- start 2024-03-11T18:33:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.771+0000 7f9ec076a740 1 -- start start 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d852c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8537960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d84e7620 con 0x5572d852cd60 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d84998c0 con 0x5572d8535570 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9ec076a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d8496df0 con 0x5572d852bff0 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45126/0 (socket says 172.21.15.2:45126) 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 -- 172.21.15.2:0/1733603850 learned_addr learned my addr 172.21.15.2:0/1733603850 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d852c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8537960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 -- 172.21.15.2:0/1733603850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 msgr2=0x5572d852c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d852c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 -- 172.21.15.2:0/1733603850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 msgr2=0x5572d8537960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8537960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6000640 1 -- 172.21.15.2:0/1733603850 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572d8413b00 con 0x5572d852cd60 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.772+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d852c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:33:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9eaf400640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 0x7f9e9c0a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9eb6000640 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009f70 tx=0x7f9ea4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9e478a50600eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9eaf400640 1 -- 172.21.15.2:0/2869263555 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9eb00f0510 con 0x7f9eb8199680 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 0x7f9e9c0a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9ea3e00640 1 -- 172.21.15.2:0/1733603850 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4004050 con 0x5572d852cd60 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9ea3e00640 1 -- 172.21.15.2:0/1733603850 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ea4005500 con 0x5572d852cd60 2024-03-11T18:33:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.773+0000 7f9ea3e00640 1 -- 172.21.15.2:0/1733603850 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4004b40 con 0x5572d852cd60 2024-03-11T18:33:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 -- 172.21.15.2:0/1733603850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 msgr2=0x5572d852bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009f70 tx=0x7f9ea4005850 comp rx=0 tx=0).stop 2024-03-11T18:33:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 0x7f9e9c0a40d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x5572d852d4c0 tx=0x7f9ea8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 -- 172.21.15.2:0/1733603850 shutdown_connections 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8537960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d852c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 --2- 172.21.15.2:0/1733603850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d852bab0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 -- 172.21.15.2:0/1733603850 >> 172.21.15.2:0/1733603850 conn(0x5572d84f34b0 msgr2=0x5572d8526ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 -- 172.21.15.2:0/1733603850 shutdown_connections 2024-03-11T18:33:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.774+0000 7f9ec076a740 1 -- 172.21.15.2:0/1733603850 wait complete. 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 -- start start 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 Processor -- start 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 0x5572d8532580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8533000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d84925d0 con 0x5572d852bff0 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d8437b50 con 0x5572d8535570 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d84377f0 con 0x5572d852cd60 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.775+0000 7f9ec076a740 10 client.0 _ll_register_callbacks cb 0x5572d844b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 0x5572d8532580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48868/0 (socket says 172.21.15.2:48868) 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 0x5572d8532580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45140/0 (socket says 172.21.15.2:45140) 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 -- 172.21.15.2:0/127400639 learned_addr learned my addr 172.21.15.2:0/127400639 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb7400640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8533000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 -- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 msgr2=0x5572d8533000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8533000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 -- 172.21.15.2:0/127400639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 msgr2=0x5572d8532580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 0x5572d8532580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 -- 172.21.15.2:0/127400639 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea4009cc0 con 0x5572d852cd60 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9eb6000640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40133a0 tx=0x7f9ea40133d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67544b3000d8fe1c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.776+0000 7f9ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4024070 con 0x5572d852cd60 2024-03-11T18:33:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572d8464cd0 con 0x5572d852cd60 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9ea40046c0 con 0x5572d852cd60 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ea4005230 con 0x5572d852cd60 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ec076a740 10 client.25507 fetch_fsmap learned FSMap version 25 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.777+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572d8490c50 con 0x5572d852cd60 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5572d8464cd0 con 0x5572d852cd60 2024-03-11T18:33:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9ea400fd60 con 0x5572d852cd60 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 10 client.25507 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 10 client.25507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 20 client.25507 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d8545400 0x5572d8547820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 4 client.25507 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:33:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572d8488ce0 con 0x5572d8545400 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.778+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d8545400 0x5572d8547820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.779+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d8545400 0x5572d8547820 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1552b9177ea0dee server_cookie=f0fe9832f17de114 in_seq=0 out_seq=0 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.779+0000 7f9ea0c00640 10 client.25507 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ea0c00640 1 -- 172.21.15.2:0/127400639 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5572d8488ce0 con 0x5572d8545400 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ea0c00640 10 client.25507 handle_command_reply: tid=0 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 client.25507 shutdown 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 msgr2=0x5572d8532ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40133a0 tx=0x7f9ea40133d0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 shutdown_connections 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d8545400 0x5572d8547820 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d8533000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d8532ac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 --2- 172.21.15.2:0/127400639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852bff0 0x5572d8532580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ea0c00640 0 client.25507 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:59.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 >> 172.21.15.2:0/127400639 conn(0x5572d84f34b0 msgr2=0x5572d8528a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 shutdown_connections 2024-03-11T18:33:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 1 -- 172.21.15.2:0/127400639 wait complete. 2024-03-11T18:33:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.780+0000 7f9ec076a740 20 client.25507 trim_cache size 0 max 16384 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.795+0000 7f9ec076a740 1 Processor -- start 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.795+0000 7f9ec076a740 1 -- start start 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d8534cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d8532e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d852f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d81dae60 con 0x5572d852cd60 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d8542050 con 0x5572d8535570 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9ec076a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d85338d0 con 0x5572d852bff0 2024-03-11T18:33:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d8534cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d852f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d8534cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48880/0 (socket says 172.21.15.2:48880) 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb6a00640 1 -- 172.21.15.2:0/742634602 learned_addr learned my addr 172.21.15.2:0/742634602 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb6000640 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d8532e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 -- 172.21.15.2:0/742634602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 msgr2=0x5572d8534cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d8534cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 -- 172.21.15.2:0/742634602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 msgr2=0x5572d8532e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d8532e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.796+0000 7f9eb7400640 1 -- 172.21.15.2:0/742634602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572d842d530 con 0x5572d8535570 2024-03-11T18:33:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.797+0000 7f9eb7400640 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d852f7e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00ed930 tx=0x7f9eb00bcbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48a7ff2d78255cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.797+0000 7f9ea3400640 1 -- 172.21.15.2:0/742634602 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00c0070 con 0x5572d8535570 2024-03-11T18:33:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.797+0000 7f9ea3400640 1 -- 172.21.15.2:0/742634602 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eb00bd460 con 0x5572d8535570 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.797+0000 7f9ea3400640 1 -- 172.21.15.2:0/742634602 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00b93d0 con 0x5572d8535570 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- 172.21.15.2:0/742634602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 msgr2=0x5572d852f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d852f7e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9eb00ed930 tx=0x7f9eb00bcbf0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- 172.21.15.2:0/742634602 shutdown_connections 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d8535570 0x5572d852f7e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d852cd60 0x5572d8532e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 --2- 172.21.15.2:0/742634602 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852bff0 0x5572d8534cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- 172.21.15.2:0/742634602 >> 172.21.15.2:0/742634602 conn(0x5572d84f34b0 msgr2=0x5572d8530e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- 172.21.15.2:0/742634602 shutdown_connections 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- 172.21.15.2:0/742634602 wait complete. 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 -- start start 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.798+0000 7f9ec076a740 1 Processor -- start 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 0x5572d853ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d8535570 0x5572d853b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d859b930 con 0x5572d8535570 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d859ba70 con 0x5572d852bff0 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572d859bbb0 con 0x5572d852cd60 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9ec076a740 10 client.0 _ll_register_callbacks cb 0x7f9ea4013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9eb7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d8535570 0x5572d853b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9eb6a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 0x5572d853ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9eb6000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48892/0 (socket says 172.21.15.2:48892) 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.799+0000 7f9eb6000640 1 -- 172.21.15.2:0/2017268100 learned_addr learned my addr 172.21.15.2:0/2017268100 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 -- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 msgr2=0x5572d853ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 0x5572d853ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 -- 172.21.15.2:0/2017268100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d8535570 msgr2=0x5572d853b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d8535570 0x5572d853b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 -- 172.21.15.2:0/2017268100 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb81a6660 con 0x5572d852cd60 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 0x5572d853ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:33:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9eb6000640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4013550 tx=0x7f9ea4009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b2b976b291e59e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4024070 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572d84b7d30 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.800+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9ea4000e40 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ea401f8b0 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 10 client.25513 fetch_fsmap learned FSMap version 25 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572d84532c0 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5572d84b7d30 con 0x5572d852cd60 2024-03-11T18:33:59.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9ea4030040 con 0x5572d852cd60 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 10 client.25513 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 10 client.25513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 20 client.25513 populate_metadata read hostname 'smithi002' 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d859cfa0 0x5572d859f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 4 client.25513 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.801+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5572d8192600 con 0x5572d859cfa0 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.802+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d859cfa0 0x5572d859f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.802+0000 7f9eb6a00640 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d859cfa0 0x5572d859f390 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f94e5f012b490ccc server_cookie=454f0c047c01147e in_seq=0 out_seq=0 2024-03-11T18:33:59.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.802+0000 7f9ea0c00640 10 client.25513 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ea0c00640 1 -- 172.21.15.2:0/2017268100 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5572d8192600 con 0x5572d859cfa0 2024-03-11T18:33:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ea0c00640 10 client.25513 handle_command_reply: tid=0 2024-03-11T18:33:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 client.25513 shutdown 2024-03-11T18:33:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 msgr2=0x5572d853b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4013550 tx=0x7f9ea4009ee0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 shutdown_connections 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5572d859cfa0 0x5572d859f390 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572d8535570 0x5572d853b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5572d852cd60 0x5572d853b380 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 --2- 172.21.15.2:0/2017268100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572d852bff0 0x5572d853ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ea0c00640 0 client.25513 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 >> 172.21.15.2:0/2017268100 conn(0x5572d84f34b0 msgr2=0x5572d85254b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 shutdown_connections 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 1 -- 172.21.15.2:0/2017268100 wait complete. 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.816+0000 7f9ec076a740 20 client.25513 trim_cache size 0 max 16384 2024-03-11T18:33:59.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:33:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:33:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:33:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:33:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 msgr2=0x7f9e9c0a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 0x7f9e9c0a40d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x5572d852d4c0 tx=0x7f9ea8005f90 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 msgr2=0x7f9eb8199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0005e30 tx=0x7f9eb00325e0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 shutdown_connections 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9eb8199680 0x7f9eb8199b00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9eb81084b0 0x7f9eb819fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9e9c0a1c10 0x7f9e9c0a40d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 --2- 172.21.15.2:0/2869263555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9eb81033e0 0x7f9eb819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 >> 172.21.15.2:0/2869263555 conn(0x7f9eb80fec40 msgr2=0x7f9eb8100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 shutdown_connections 2024-03-11T18:33:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:33:59.817+0000 7f9eb7e00640 1 -- 172.21.15.2:0/2869263555 wait complete. 2024-03-11T18:34:00.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-mon[31328]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-11T18:34:00.306 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:34:00.081+0000 7fae9c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:34:00.306 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:34:00.081+0000 7fae9c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:34:00.306 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:34:00.081+0000 7fae9c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:34:00.306 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:34:00.081+0000 7fae9c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:34:00.306 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:34:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:34:00.081+0000 7fae9c600700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-11T18:34:00.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:00 smithi165 ceph-mon[36634]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-11T18:34:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:00 smithi023 ceph-mon[36406]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-11T18:34:02.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:02 smithi165 ceph-mon[36634]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-11T18:34:02.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:02 smithi165 ceph-mon[36634]: mgrmap e26: x(active, since 18s), standbys: y, z 2024-03-11T18:34:02.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:02 smithi023 ceph-mon[36406]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-11T18:34:02.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:02 smithi023 ceph-mon[36406]: mgrmap e26: x(active, since 18s), standbys: y, z 2024-03-11T18:34:02.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:02 smithi002 ceph-mon[31328]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-11T18:34:02.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:02 smithi002 ceph-mon[31328]: mgrmap e26: x(active, since 18s), standbys: y, z 2024-03-11T18:34:04.378 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:04 smithi023 ceph-mon[36406]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:04.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:04 smithi165 ceph-mon[36634]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:04 smithi002 ceph-mon[31328]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:06.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:06 smithi165 ceph-mon[36634]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T18:34:06.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:06 smithi023 ceph-mon[36406]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T18:34:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:06 smithi002 ceph-mon[31328]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-03-11T18:34:08.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:08 smithi165 ceph-mon[36634]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-11T18:34:08.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:08 smithi023 ceph-mon[36406]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-11T18:34:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:08 smithi002 ceph-mon[31328]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-03-11T18:34:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:10 smithi165 ceph-mon[36634]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:10.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:10 smithi023 ceph-mon[36406]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:10.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:10 smithi002 ceph-mon[31328]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:12.385 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:12 smithi002 ceph-mon[31328]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-11T18:34:12.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:12 smithi165 ceph-mon[36634]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-11T18:34:12.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:12 smithi023 ceph-mon[36406]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-03-11T18:34:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 -- 172.21.15.2:0/3936369511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b0073ed0 msgr2=0x7fb0b010d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/3936369511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b0073ed0 0x7fb0b010d260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009900 tx=0x7fb09c02bcd0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 -- 172.21.15.2:0/3936369511 shutdown_connections 2024-03-11T18:34:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/3936369511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b010d7a0 0x7fb0b010fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/3936369511 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b0073ed0 0x7fb0b010d260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/3936369511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b0073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 -- 172.21.15.2:0/3936369511 >> 172.21.15.2:0/3936369511 conn(0x7fb0b00fca30 msgr2=0x7fb0b00fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 -- 172.21.15.2:0/3936369511 shutdown_connections 2024-03-11T18:34:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.975+0000 7fb0b5a00640 1 -- 172.21.15.2:0/3936369511 wait complete. 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.976+0000 7fb0b5a00640 1 Processor -- start 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.976+0000 7fb0b5a00640 1 -- start start 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b0073ed0 0x7fb0b019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 0x7fb0b0199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0199c70 con 0x7fb0b00735b0 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0199de0 con 0x7fb0b0073ed0 2024-03-11T18:34:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b0199fa0 con 0x7fb0b010d7a0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0b5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 0x7fb0b0199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50694/0 (socket says 172.21.15.2:50694) 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0afe00640 1 -- 172.21.15.2:0/2514028106 learned_addr learned my addr 172.21.15.2:0/2514028106 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.977+0000 7fb0af400640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b0073ed0 0x7fb0b019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 -- 172.21.15.2:0/2514028106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b0073ed0 msgr2=0x7fb0b019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b0073ed0 0x7fb0b019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 -- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 msgr2=0x7fb0b019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 -- 172.21.15.2:0/2514028106 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c009560 con 0x7fb0b010d7a0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0afe00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 0x7fb0b0199620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800d680 tx=0x7fb0a800db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0acc00640 1 -- 172.21.15.2:0/2514028106 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8005220 con 0x7fb0b010d7a0 2024-03-11T18:34:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0acc00640 1 -- 172.21.15.2:0/2514028106 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0a8009d00 con 0x7fb0b010d7a0 2024-03-11T18:34:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b019a280 con 0x7fb0b010d7a0 2024-03-11T18:34:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.978+0000 7fb0acc00640 1 -- 172.21.15.2:0/2514028106 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8004ea0 con 0x7fb0b010d7a0 2024-03-11T18:34:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.979+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b01a6510 con 0x7fb0b010d7a0 2024-03-11T18:34:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0acc00640 1 -- 172.21.15.2:0/2514028106 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 26) v1 ==== 136511+0+0 (secure 0 0 0) 0x7fb0a8009820 con 0x7fb0b010d7a0 2024-03-11T18:34:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 Processor -- start 2024-03-11T18:34:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 -- start start 2024-03-11T18:34:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 0x55a815cefab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815cfb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815cab620 con 0x55a815cf0d60 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815c5d8c0 con 0x55a815cf9570 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.982+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815c5adf0 con 0x55a815cefff0 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0af400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 0x55a815cefab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57130/0 (socket says 172.21.15.2:57130) 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 -- 172.21.15.2:0/706092386 learned_addr learned my addr 172.21.15.2:0/706092386 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0af400640 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815cfb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 -- 172.21.15.2:0/706092386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 msgr2=0x55a815cfb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815cfb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 -- 172.21.15.2:0/706092386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 msgr2=0x55a815cefab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 0x55a815cefab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 -- 172.21.15.2:0/706092386 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a815bd7b00 con 0x55a815cefff0 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0af400640 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 0x55a815cefab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.983+0000 7fb0afe00640 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000f8d0 tx=0x7fb0a000fd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5357e818932efcab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0a5600640 1 -- 172.21.15.2:0/706092386 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0011c20 con 0x55a815cefff0 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0a5600640 1 -- 172.21.15.2:0/706092386 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0a0011d80 con 0x55a815cefff0 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0a5600640 1 -- 172.21.15.2:0/706092386 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00187b0 con 0x55a815cefff0 2024-03-11T18:34:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0acc00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 0x7fb0940a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0b9c65740 1 -- 172.21.15.2:0/706092386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 msgr2=0x55a815cf0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0acc00640 1 -- 172.21.15.2:0/2514028106 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb0a80cd660 con 0x7fb0b010d7a0 2024-03-11T18:34:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.984+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000f8d0 tx=0x7fb0a000fd90 comp rx=0 tx=0).stop 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0af400640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 0x7fb0940a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 -- 172.21.15.2:0/706092386 shutdown_connections 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815cfb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815cf0470 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/706092386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf0d60 0x55a815cefab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 -- 172.21.15.2:0/706092386 >> 172.21.15.2:0/706092386 conn(0x55a815cb74b0 msgr2=0x55a815ceace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 -- 172.21.15.2:0/706092386 shutdown_connections 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0b9c65740 1 -- 172.21.15.2:0/706092386 wait complete. 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.985+0000 7fb0af400640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 0x7fb0940a41e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb09c005ab0 tx=0x7fb09c005e20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.986+0000 7fb0b9c65740 1 -- start start 2024-03-11T18:34:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.986+0000 7fb0b9c65740 1 Processor -- start 2024-03-11T18:34:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815cf6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815c565d0 con 0x55a815cefff0 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815bfbb50 con 0x55a815cf0d60 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815bfb7f0 con 0x55a815cf9570 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.987+0000 7fb0b9c65740 10 client.0 _ll_register_callbacks cb 0x55a815c0f550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57140/0 (socket says 172.21.15.2:57140) 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 -- 172.21.15.2:0/2147742593 learned_addr learned my addr 172.21.15.2:0/2147742593 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0afe00640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 -- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 msgr2=0x55a815cf6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815cf6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 -- 172.21.15.2:0/2147742593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 msgr2=0x55a815cf6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 -- 172.21.15.2:0/2147742593 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a000f530 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb0b5000640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8096150 tx=0x7fb0a8005380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43641a75b47b1230 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.988+0000 7fb09a000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80a23e0 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a815c28cd0 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb0a8005580 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0a80cd200 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 10 client.25531 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a815c54c50 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a815c28cd0 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb0a8099140 con 0x55a815cf9570 2024-03-11T18:34:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 10 client.25531 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 10 client.25531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.989+0000 7fb0b9c65740 20 client.25531 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.990+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d1a550 0x55a815d1c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.990+0000 7fb0b9c65740 4 client.25531 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.990+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a815c4cce0 con 0x55a815d1a550 2024-03-11T18:34:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.990+0000 7fb0afe00640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d1a550 0x55a815d1c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.991+0000 7fb0afe00640 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d1a550 0x55a815d1c940 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=613297097beb3131 server_cookie=9c412834026aa6da in_seq=0 out_seq=0 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.991+0000 7fb09a000640 10 client.25531 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.991+0000 7fb09a000640 1 -- 172.21.15.2:0/2147742593 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a815c4cce0 con 0x55a815d1a550 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.991+0000 7fb09a000640 10 client.25531 handle_command_reply: tid=0 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.991+0000 7fb0b9c65740 1 client.25531 shutdown 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 msgr2=0x55a815cf6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8096150 tx=0x7fb0a8005380 comp rx=0 tx=0).stop 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 shutdown_connections 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d1a550 0x55a815d1c940 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf9570 0x55a815cf6fc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815cf6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/2147742593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb09a000640 0 client.25531 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 >> 172.21.15.2:0/2147742593 conn(0x55a815cb74b0 msgr2=0x55a815ceca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 shutdown_connections 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 1 -- 172.21.15.2:0/2147742593 wait complete. 2024-03-11T18:34:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:12.992+0000 7fb0b9c65740 20 client.25531 trim_cache size 0 max 16384 2024-03-11T18:34:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 Processor -- start 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 -- start start 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815d00390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a81599ee60 con 0x55a815cefff0 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815ceafe0 con 0x55a815cf9570 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.007+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815ce6610 con 0x55a815cf0d60 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57144/0 (socket says 172.21.15.2:57144) 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 -- 172.21.15.2:0/997614540 learned_addr learned my addr 172.21.15.2:0/997614540 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0b5000640 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815d00390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 -- 172.21.15.2:0/997614540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 msgr2=0x55a815d00390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815d00390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 -- 172.21.15.2:0/997614540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 msgr2=0x55a815cf8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 -- 172.21.15.2:0/997614540 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a815bf1530 con 0x55a815cf0d60 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.008+0000 7fb0af400640 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb09c035000 tx=0x7fb09c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21a0960ce561ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.009+0000 7fb0a4c00640 1 -- 172.21.15.2:0/997614540 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c048040 con 0x55a815cf0d60 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.009+0000 7fb0a4c00640 1 -- 172.21.15.2:0/997614540 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09c02cd60 con 0x55a815cf0d60 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.009+0000 7fb0a4c00640 1 -- 172.21.15.2:0/997614540 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c0035f0 con 0x55a815cf0d60 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- 172.21.15.2:0/997614540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 msgr2=0x55a815cf6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb09c035000 tx=0x7fb09c009260 comp rx=0 tx=0).stop 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- 172.21.15.2:0/997614540 shutdown_connections 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf9570 0x55a815d00390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cf0d60 0x55a815cf6de0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/997614540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cefff0 0x55a815cf8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- 172.21.15.2:0/997614540 >> 172.21.15.2:0/997614540 conn(0x55a815cb74b0 msgr2=0x55a815cf1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- 172.21.15.2:0/997614540 shutdown_connections 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- 172.21.15.2:0/997614540 wait complete. 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 -- start start 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.010+0000 7fb0b9c65740 1 Processor -- start 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815d008a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 0x55a815cfc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815cfd050 con 0x55a815cf9570 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815cfd1c0 con 0x55a815cf0d60 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a815cfd380 con 0x55a815cefff0 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b9c65740 10 client.0 _ll_register_callbacks cb 0x7fb0a8009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57154/0 (socket says 172.21.15.2:57154) 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0afe00640 1 -- 172.21.15.2:0/4177381302 learned_addr learned my addr 172.21.15.2:0/4177381302 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0b5000640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 0x55a815cfc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.011+0000 7fb0af400640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815d008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 -- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 msgr2=0x55a815d008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815d008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 -- 172.21.15.2:0/4177381302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 msgr2=0x55a815cfc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 0x55a815cfc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 -- 172.21.15.2:0/4177381302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b01108c0 con 0x55a815cefff0 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0b5000640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 0x55a815cfc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0af400640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815d008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0afe00640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000f80 tx=0x7fb0a00183d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4636389e0acd80e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb09a000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0029040 con 0x55a815cefff0 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb0a0003970 con 0x55a815cefff0 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.012+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a815c7bd30 con 0x55a815cefff0 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0a0003b30 con 0x55a815cefff0 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 10 client.25537 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a815c172c0 con 0x55a815cefff0 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a815c7bd30 con 0x55a815cefff0 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb0a0021040 con 0x55a815cefff0 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 10 client.25537 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 10 client.25537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 20 client.25537 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d71db0 0x55a815d741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 4 client.25537 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:34:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.013+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a815956600 con 0x55a815d71db0 2024-03-11T18:34:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.014+0000 7fb0af400640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d71db0 0x55a815d741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.014+0000 7fb0af400640 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d71db0 0x55a815d741d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5427eb6140fe4b9b server_cookie=3ffdf75c74fe9b2d in_seq=0 out_seq=0 2024-03-11T18:34:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.014+0000 7fb09a000640 10 client.25537 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb09a000640 1 -- 172.21.15.2:0/4177381302 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a815956600 con 0x55a815d71db0 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb09a000640 10 client.25537 handle_command_reply: tid=0 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 client.25537 shutdown 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 msgr2=0x55a815d00360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000f80 tx=0x7fb0a00183d0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 shutdown_connections 2024-03-11T18:34:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a815d71db0 0x55a815d741d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a815cf9570 0x55a815cfc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a815cf0d60 0x55a815d008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb09a000640 0 client.25537 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 --2- 172.21.15.2:0/4177381302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a815cefff0 0x55a815d00360 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 >> 172.21.15.2:0/4177381302 conn(0x55a815cb74b0 msgr2=0x55a815cef510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 shutdown_connections 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.022+0000 7fb0b9c65740 1 -- 172.21.15.2:0/4177381302 wait complete. 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b9c65740 20 client.25537 trim_cache size 0 max 16384 2024-03-11T18:34:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 msgr2=0x7fb0940a41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 0x7fb0940a41e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb09c005ab0 tx=0x7fb09c005e20 comp rx=0 tx=0).stop 2024-03-11T18:34:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 msgr2=0x7fb0b0199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 0x7fb0b0199620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800d680 tx=0x7fb0a800db40 comp rx=0 tx=0).stop 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 shutdown_connections 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb0b010d7a0 0x7fb0b0199620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0b0073ed0 0x7fb0b019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb0940a1d20 0x7fb0940a41e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 --2- 172.21.15.2:0/2514028106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b00735b0 0x7fb0b019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.023+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 >> 172.21.15.2:0/2514028106 conn(0x7fb0b00fca30 msgr2=0x7fb0b00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.024+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 shutdown_connections 2024-03-11T18:34:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:13.024+0000 7fb0b5a00640 1 -- 172.21.15.2:0/2514028106 wait complete. 2024-03-11T18:34:14.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:14 smithi023 ceph-mon[36406]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:14.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:14 smithi165 ceph-mon[36634]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:14.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:14 smithi002 ceph-mon[31328]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-11T18:34:16.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:16 smithi165 ceph-mon[36634]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-11T18:34:16.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:16 smithi023 ceph-mon[36406]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-11T18:34:16.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:16 smithi002 ceph-mon[31328]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 29 op/s 2024-03-11T18:34:18.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:18 smithi165 ceph-mon[36634]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:18.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:18 smithi023 ceph-mon[36406]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:18 smithi002 ceph-mon[31328]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:20 smithi165 ceph-mon[36634]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:20 smithi023 ceph-mon[36406]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:20.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:20 smithi002 ceph-mon[31328]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:22 smithi165 ceph-mon[36634]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-11T18:34:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:22 smithi023 ceph-mon[36406]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-11T18:34:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:22 smithi002 ceph-mon[31328]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-11T18:34:24.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:24 smithi165 ceph-mon[36634]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:24 smithi023 ceph-mon[36406]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:24.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:24 smithi002 ceph-mon[31328]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 -- 172.21.15.2:0/2269387498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 msgr2=0x7f7a3c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2269387498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c1015b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009900 tx=0x7f7a2c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 -- 172.21.15.2:0/2269387498 shutdown_connections 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2269387498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2269387498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c101b80 0x7f7a3c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2269387498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c1015b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 -- 172.21.15.2:0/2269387498 >> 172.21.15.2:0/2269387498 conn(0x7f7a3c0fca30 msgr2=0x7f7a3c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 -- 172.21.15.2:0/2269387498 shutdown_connections 2024-03-11T18:34:26.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.167+0000 7f7a3be00640 1 -- 172.21.15.2:0/2269387498 wait complete. 2024-03-11T18:34:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.168+0000 7f7a3be00640 1 Processor -- start 2024-03-11T18:34:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.168+0000 7f7a3be00640 1 -- start start 2024-03-11T18:34:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c199620 0x7f7a3c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c19a010 con 0x7f7a3c1062a0 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c19a180 con 0x7f7a3c199620 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c19a340 con 0x7f7a3c1011d0 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34492/0 (socket says 172.21.15.2:34492) 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a33e00640 1 -- 172.21.15.2:0/2315069828 learned_addr learned my addr 172.21.15.2:0/2315069828 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.169+0000 7f7a3b400640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c199620 0x7f7a3c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 -- 172.21.15.2:0/2315069828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 msgr2=0x7f7a3c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 -- 172.21.15.2:0/2315069828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c199620 msgr2=0x7f7a3c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c199620 0x7f7a3c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 -- 172.21.15.2:0/2315069828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a34009840 con 0x7f7a3c1062a0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33e00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e9f0 tx=0x7f7a2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33400640 1 -- 172.21.15.2:0/2315069828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a240096c0 con 0x7f7a3c1062a0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c009560 con 0x7f7a3c1062a0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.170+0000 7f7a33400640 1 -- 172.21.15.2:0/2315069828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a24005500 con 0x7f7a3c1062a0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.171+0000 7f7a33400640 1 -- 172.21.15.2:0/2315069828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24010430 con 0x7f7a3c1062a0 2024-03-11T18:34:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.171+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c1a66b0 con 0x7f7a3c1062a0 2024-03-11T18:34:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.174+0000 7f7a33400640 1 -- 172.21.15.2:0/2315069828 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 136511+0+0 (secure 0 0 0) 0x7f7a24005020 con 0x7f7a3c1062a0 2024-03-11T18:34:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.174+0000 7f7a4497a740 1 Processor -- start 2024-03-11T18:34:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.174+0000 7f7a4497a740 1 -- start start 2024-03-11T18:34:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96debab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96dec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96df7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96da7620 con 0x555c96debff0 2024-03-11T18:34:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96d598c0 con 0x555c96df5570 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a4497a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96d56df0 con 0x555c96decd60 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96dec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96dec470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34494/0 (socket says 172.21.15.2:34494) 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a3aa00640 1 -- 172.21.15.2:0/3793500635 learned_addr learned my addr 172.21.15.2:0/3793500635 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a3b400640 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96debab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.175+0000 7f7a33e00640 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96df7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 -- 172.21.15.2:0/3793500635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 msgr2=0x555c96df7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96df7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 -- 172.21.15.2:0/3793500635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 msgr2=0x555c96dec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96dec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 -- 172.21.15.2:0/3793500635 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c96cd3b00 con 0x555c96decd60 2024-03-11T18:34:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a33400640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 0x7f7a200a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3b400640 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96debab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011540 tx=0x7f7a34011a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b01bc9fb222340 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a33400640 1 -- 172.21.15.2:0/2315069828 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7a24014070 con 0x7f7a3c1062a0 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 0x7f7a200a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.176+0000 7f7a2be00640 1 -- 172.21.15.2:0/3793500635 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34004d20 con 0x555c96decd60 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.177+0000 7f7a2be00640 1 -- 172.21.15.2:0/3793500635 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a34004e80 con 0x555c96decd60 2024-03-11T18:34:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.177+0000 7f7a2be00640 1 -- 172.21.15.2:0/3793500635 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400f7c0 con 0x555c96decd60 2024-03-11T18:34:26.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- 172.21.15.2:0/3793500635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 msgr2=0x555c96debab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 0x7f7a200a4160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x555c96ded4c0 tx=0x7f7a2c005e40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96debab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011540 tx=0x7f7a34011a00 comp rx=0 tx=0).stop 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- 172.21.15.2:0/3793500635 shutdown_connections 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96df7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96dec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 --2- 172.21.15.2:0/3793500635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96debab0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- 172.21.15.2:0/3793500635 >> 172.21.15.2:0/3793500635 conn(0x555c96db34b0 msgr2=0x555c96de6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- 172.21.15.2:0/3793500635 shutdown_connections 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- 172.21.15.2:0/3793500635 wait complete. 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.178+0000 7f7a4497a740 1 -- start start 2024-03-11T18:34:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 Processor -- start 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 0x555c96df2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 0x555c96df3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96d525d0 con 0x555c96df5570 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96cf7b50 con 0x555c96decd60 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.179+0000 7f7a4497a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96cf77f0 con 0x555c96debff0 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a4497a740 10 client.0 _ll_register_callbacks cb 0x555c96d0b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 0x555c96df3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57276/0 (socket says 172.21.15.2:57276) 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3aa00640 1 -- 172.21.15.2:0/4220865458 learned_addr learned my addr 172.21.15.2:0/4220865458 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3b400640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 0x555c96df3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34504/0 (socket says 172.21.15.2:34504) 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 0x555c96df2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 -- 172.21.15.2:0/4220865458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 msgr2=0x555c96df2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 -- 172.21.15.2:0/4220865458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 msgr2=0x555c96df3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 0x555c96df3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a33e00640 1 -- 172.21.15.2:0/4220865458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a34009d80 con 0x555c96decd60 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.180+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a33e00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 0x555c96df2b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a240cb3b0 tx=0x7f7a240963a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a0ee7cf28ef9375 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a28c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2409a300 con 0x555c96decd60 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c96d24cd0 con 0x555c96decd60 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7a24005670 con 0x555c96decd60 2024-03-11T18:34:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a2409a120 con 0x555c96decd60 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a4497a740 10 client.25538 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555c96d50c50 con 0x555c96decd60 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.181+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555c96d24cd0 con 0x555c96decd60 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a2401f060 con 0x555c96decd60 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 10 client.25538 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 10 client.25538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 20 client.25538 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e05690 0x555c96e07ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 4 client.25538 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c96d48ce0 con 0x555c96e05690 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.182+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e05690 0x555c96e07ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.183+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e05690 0x555c96e07ab0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f75a4f1bb69a01f4 server_cookie=f4aa4d41dec6e82e in_seq=0 out_seq=0 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.183+0000 7f7a28c00640 10 client.25538 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.183+0000 7f7a28c00640 1 -- 172.21.15.2:0/4220865458 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c96d48ce0 con 0x555c96e05690 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.183+0000 7f7a28c00640 10 client.25538 handle_command_reply: tid=0 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 client.25538 shutdown 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 msgr2=0x555c96df2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 0x555c96df2b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7a240cb3b0 tx=0x7f7a240963a0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 shutdown_connections 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e05690 0x555c96e07ab0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96df5570 0x555c96df3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a28c00640 0 client.25538 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96decd60 0x555c96df2b50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 --2- 172.21.15.2:0/4220865458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96debff0 0x555c96df2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 >> 172.21.15.2:0/4220865458 conn(0x555c96db34b0 msgr2=0x555c96de8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 shutdown_connections 2024-03-11T18:34:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 1 -- 172.21.15.2:0/4220865458 wait complete. 2024-03-11T18:34:26.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.184+0000 7f7a4497a740 20 client.25538 trim_cache size 0 max 16384 2024-03-11T18:34:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.211+0000 7f7a4497a740 1 Processor -- start 2024-03-11T18:34:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 -- start start 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96df2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96dfc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96a9ae60 con 0x555c96debff0 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96d24d00 con 0x555c96df5570 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a4497a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96df7d30 con 0x555c96decd60 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34510/0 (socket says 172.21.15.2:34510) 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a3aa00640 1 -- 172.21.15.2:0/2147261721 learned_addr learned my addr 172.21.15.2:0/2147261721 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.212+0000 7f7a3b400640 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96dfc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a33e00640 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96df2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 -- 172.21.15.2:0/2147261721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 msgr2=0x555c96df2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96df2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 -- 172.21.15.2:0/2147261721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 msgr2=0x555c96dfc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96dfc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 -- 172.21.15.2:0/2147261721 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c96ced530 con 0x555c96debff0 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.213+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c000c00 tx=0x7f7a2c045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4578c304e41156 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a2b400640 1 -- 172.21.15.2:0/2147261721 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0041c0 con 0x555c96debff0 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a2b400640 1 -- 172.21.15.2:0/2147261721 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a2c055880 con 0x555c96debff0 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a2b400640 1 -- 172.21.15.2:0/2147261721 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c003d70 con 0x555c96debff0 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 -- 172.21.15.2:0/2147261721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 msgr2=0x555c96df4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c000c00 tx=0x7f7a2c045000 comp rx=0 tx=0).stop 2024-03-11T18:34:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 -- 172.21.15.2:0/2147261721 shutdown_connections 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96df5570 0x555c96dfc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96decd60 0x555c96df2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.214+0000 7f7a4497a740 1 --2- 172.21.15.2:0/2147261721 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96debff0 0x555c96df4cd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.215+0000 7f7a4497a740 1 -- 172.21.15.2:0/2147261721 >> 172.21.15.2:0/2147261721 conn(0x555c96db34b0 msgr2=0x555c96de6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.215+0000 7f7a4497a740 1 -- 172.21.15.2:0/2147261721 shutdown_connections 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.215+0000 7f7a4497a740 1 -- 172.21.15.2:0/2147261721 wait complete. 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.215+0000 7f7a4497a740 1 -- start start 2024-03-11T18:34:26.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.215+0000 7f7a4497a740 1 Processor -- start 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96debff0 0x555c96def140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96df5570 0x555c96defbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96dfccc0 con 0x555c96decd60 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96e5bb90 con 0x555c96debff0 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c96e5bcd0 con 0x555c96df5570 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a4497a740 10 client.0 _ll_register_callbacks cb 0x7f7a24004990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34512/0 (socket says 172.21.15.2:34512) 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 -- 172.21.15.2:0/1935884880 learned_addr learned my addr 172.21.15.2:0/1935884880 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:26.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a3b400640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96df5570 0x555c96defbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96debff0 0x555c96def140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 -- 172.21.15.2:0/1935884880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96df5570 msgr2=0x555c96defbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96df5570 0x555c96defbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 -- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96debff0 msgr2=0x555c96def140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96debff0 0x555c96def140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.216+0000 7f7a33e00640 1 -- 172.21.15.2:0/1935884880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c10e680 con 0x555c96decd60 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a33e00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7a2409a4c0 tx=0x7f7a240b3420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ae97ff8441736a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a28c00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24004800 con 0x555c96decd60 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7a24005670 con 0x555c96decd60 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c96d77d30 con 0x555c96decd60 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a2409a300 con 0x555c96decd60 2024-03-11T18:34:26.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a4497a740 10 client.15453 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.217+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555c96d132c0 con 0x555c96decd60 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555c96d77d30 con 0x555c96decd60 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a2401f030 con 0x555c96decd60 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 10 client.15453 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 10 client.15453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 20 client.15453 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e5d030 0x555c96e5f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 4 client.15453 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555c96a52600 con 0x555c96e5d030 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.218+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e5d030 0x555c96e5f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.219+0000 7f7a3aa00640 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e5d030 0x555c96e5f420 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26657baa879ea978 server_cookie=9742db6bef9e17eb in_seq=0 out_seq=0 2024-03-11T18:34:26.227 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.219+0000 7f7a28c00640 10 client.15453 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.226+0000 7f7a28c00640 1 -- 172.21.15.2:0/1935884880 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555c96a52600 con 0x555c96e5d030 2024-03-11T18:34:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a28c00640 10 client.15453 handle_command_reply: tid=0 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 client.15453 shutdown 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 msgr2=0x555c96def680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7a2409a4c0 tx=0x7f7a240b3420 comp rx=0 tx=0).stop 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 shutdown_connections 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x555c96e5d030 0x555c96e5f420 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x555c96df5570 0x555c96defbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x555c96decd60 0x555c96def680 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 --2- 172.21.15.2:0/1935884880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96debff0 0x555c96def140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a28c00640 0 client.15453 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 >> 172.21.15.2:0/1935884880 conn(0x555c96db34b0 msgr2=0x555c96df0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 shutdown_connections 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 1 -- 172.21.15.2:0/1935884880 wait complete. 2024-03-11T18:34:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.227+0000 7f7a4497a740 20 client.15453 trim_cache size 0 max 16384 2024-03-11T18:34:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:34:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:34:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:34:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 msgr2=0x7f7a200a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 0x7f7a200a4160 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x555c96ded4c0 tx=0x7f7a2c005e40 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 msgr2=0x7f7a3c19f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7a2400e9f0 tx=0x7f7a2400edf0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 shutdown_connections 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a3c199620 0x7f7a3c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7a200a1ca0 0x7f7a200a4160 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a3c1062a0 0x7f7a3c19f9b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 --2- 172.21.15.2:0/2315069828 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a3c1011d0 0x7f7a3c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 >> 172.21.15.2:0/2315069828 conn(0x7f7a3c0fca30 msgr2=0x7f7a3c0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 shutdown_connections 2024-03-11T18:34:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:26.228+0000 7f7a3be00640 1 -- 172.21.15.2:0/2315069828 wait complete. 2024-03-11T18:34:26.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:26 smithi023 ceph-mon[36406]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:26.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:26 smithi165 ceph-mon[36634]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:26.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:26 smithi002 ceph-mon[31328]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:28.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:28 smithi165 ceph-mon[36634]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-11T18:34:28.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:28 smithi023 ceph-mon[36406]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-11T18:34:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:28 smithi002 ceph-mon[31328]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-11T18:34:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:30 smithi165 ceph-mon[36634]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T18:34:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:30 smithi023 ceph-mon[36406]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T18:34:30.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:30 smithi002 ceph-mon[31328]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-11T18:34:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:32 smithi002 ceph-mon[31328]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-11T18:34:32.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:32 smithi023 ceph-mon[36406]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-11T18:34:32.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:32 smithi165 ceph-mon[36634]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-11T18:34:34.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:34 smithi165 ceph-mon[36634]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-11T18:34:34.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:34 smithi023 ceph-mon[36406]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-11T18:34:34.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:34 smithi002 ceph-mon[31328]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-03-11T18:34:36.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:36 smithi023 ceph-mon[36406]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:36.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:36 smithi165 ceph-mon[36634]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:36.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:36 smithi002 ceph-mon[31328]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.5 KiB/s wr, 9 op/s 2024-03-11T18:34:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:38 smithi165 ceph-mon[36634]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:38.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:38 smithi023 ceph-mon[36406]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:38.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:38 smithi002 ceph-mon[31328]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T18:34:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 -- 172.21.15.2:0/1714530283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc101b80 msgr2=0x7f6ddc105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 --2- 172.21.15.2:0/1714530283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc101b80 0x7f6ddc105bc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009930 tx=0x7f6dc802bc20 comp rx=0 tx=0).stop 2024-03-11T18:34:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 -- 172.21.15.2:0/1714530283 shutdown_connections 2024-03-11T18:34:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 --2- 172.21.15.2:0/1714530283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 0x7f6ddc10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 --2- 172.21.15.2:0/1714530283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc101b80 0x7f6ddc105bc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 --2- 172.21.15.2:0/1714530283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 -- 172.21.15.2:0/1714530283 >> 172.21.15.2:0/1714530283 conn(0x7f6ddc0fca90 msgr2=0x7f6ddc0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 -- 172.21.15.2:0/1714530283 shutdown_connections 2024-03-11T18:34:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.353+0000 7f6de2e00640 1 -- 172.21.15.2:0/1714530283 wait complete. 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.354+0000 7f6de2e00640 1 Processor -- start 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.354+0000 7f6de2e00640 1 -- start start 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 0x7f6ddc0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 0x7f6ddc1a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1a5dd0 con 0x7f6ddc1062a0 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1a5f40 con 0x7f6ddc073f00 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1a6100 con 0x7f6ddc1011d0 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40630/0 (socket says 172.21.15.2:40630) 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de1a00640 1 -- 172.21.15.2:0/2170494885 learned_addr learned my addr 172.21.15.2:0/2170494885 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de2400640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 0x7f6ddc1a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.355+0000 7f6de1000640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 0x7f6ddc0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 -- 172.21.15.2:0/2170494885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 msgr2=0x7f6ddc1a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 0x7f6ddc1a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 -- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 msgr2=0x7f6ddc0739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 0x7f6ddc0739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 -- 172.21.15.2:0/2170494885 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8009560 con 0x7f6ddc1011d0 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de2400640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 0x7f6ddc1a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:34:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000d950 tx=0x7f6dd000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6dda000640 1 -- 172.21.15.2:0/2170494885 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0009c30 con 0x7f6ddc1011d0 2024-03-11T18:34:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.356+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc1a63e0 con 0x7f6ddc1011d0 2024-03-11T18:34:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.357+0000 7f6dda000640 1 -- 172.21.15.2:0/2170494885 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd0009d90 con 0x7f6ddc1011d0 2024-03-11T18:34:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.357+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc1a6910 con 0x7f6ddc1011d0 2024-03-11T18:34:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.357+0000 7f6dda000640 1 -- 172.21.15.2:0/2170494885 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000f7e0 con 0x7f6ddc1011d0 2024-03-11T18:34:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.360+0000 7f6dda000640 1 -- 172.21.15.2:0/2170494885 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 26) v1 ==== 136511+0+0 (secure 0 0 0) 0x7f6dd000f940 con 0x7f6ddc1011d0 2024-03-11T18:34:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.361+0000 7f6dda000640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 0x7f6dc40a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.361+0000 7f6de71e9740 1 Processor -- start 2024-03-11T18:34:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.361+0000 7f6dda000640 1 -- 172.21.15.2:0/2170494885 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6dd00cdc00 con 0x7f6ddc1011d0 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.361+0000 7f6de71e9740 1 -- start start 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de1000640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 0x7f6dc40a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf2cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf2d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcee8620 con 0x558abcf2cff0 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce9a8c0 con 0x558abcf2dd60 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de71e9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce97df0 con 0x558abcf36570 2024-03-11T18:34:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf2cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de2400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40646/0 (socket says 172.21.15.2:40646) 2024-03-11T18:34:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf2cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:41354/0 (socket says 172.21.15.2:41354) 2024-03-11T18:34:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.362+0000 7f6de2400640 1 -- 172.21.15.2:0/4000184735 learned_addr learned my addr 172.21.15.2:0/4000184735 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 -- 172.21.15.2:0/4000184735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 msgr2=0x558abcf2cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf2cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 -- 172.21.15.2:0/4000184735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 msgr2=0x558abcf2d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf2d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 -- 172.21.15.2:0/4000184735 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558abce14b00 con 0x558abcf36570 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de2400640 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4011790 tx=0x7f6dd4011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2389ebf0599aab20 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.363+0000 7f6de1000640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 0x7f6dc40a4210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6dc802c160 tx=0x7f6dc80046e0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.364+0000 7f6dcea00640 1 -- 172.21.15.2:0/4000184735 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4011ea0 con 0x558abcf36570 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.364+0000 7f6dcea00640 1 -- 172.21.15.2:0/4000184735 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd4004d20 con 0x558abcf36570 2024-03-11T18:34:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.364+0000 7f6dcea00640 1 -- 172.21.15.2:0/4000184735 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4004290 con 0x558abcf36570 2024-03-11T18:34:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 -- 172.21.15.2:0/4000184735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 msgr2=0x558abcf38960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4011790 tx=0x7f6dd4011c50 comp rx=0 tx=0).stop 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 -- 172.21.15.2:0/4000184735 shutdown_connections 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf36570 0x558abcf38960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf2d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 --2- 172.21.15.2:0/4000184735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf2cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 -- 172.21.15.2:0/4000184735 >> 172.21.15.2:0/4000184735 conn(0x558abcef44b0 msgr2=0x558abcf27ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 -- 172.21.15.2:0/4000184735 shutdown_connections 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.365+0000 7f6de71e9740 1 -- 172.21.15.2:0/4000184735 wait complete. 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.366+0000 7f6de71e9740 1 -- start start 2024-03-11T18:34:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.366+0000 7f6de71e9740 1 Processor -- start 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2dd60 0x558abcf33aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 0x558abcf33fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce935d0 con 0x558abcf2cff0 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce38b50 con 0x558abcf36570 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abce387f0 con 0x558abcf2dd60 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de71e9740 10 client.0 _ll_register_callbacks cb 0x558abce4c550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de2400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 0x558abcf33fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48266/0 (socket says 172.21.15.2:48266) 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.367+0000 7f6de1a00640 1 -- 172.21.15.2:0/563741618 learned_addr learned my addr 172.21.15.2:0/563741618 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 -- 172.21.15.2:0/563741618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2dd60 msgr2=0x558abcf33aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2dd60 0x558abcf33aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 -- 172.21.15.2:0/563741618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 msgr2=0x558abcf33560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 -- 172.21.15.2:0/563741618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd4011440 con 0x558abcf36570 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558abce65cd0 con 0x558abcf36570 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6de2400640 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 0x558abcf33fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4011760 tx=0x7f6dd4003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a219a114f8d90289 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6dc3400640 10 client.25556 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6dc3400640 1 -- 172.21.15.2:0/563741618 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4011ea0 con 0x558abcf36570 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.368+0000 7f6dc3400640 1 -- 172.21.15.2:0/563741618 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6dd4004c40 con 0x558abcf36570 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6dc3400640 1 -- 172.21.15.2:0/563741618 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dd4004e20 con 0x558abcf36570 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 10 client.25556 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x558abce91c50 con 0x558abcf36570 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6dc3400640 1 -- 172.21.15.2:0/563741618 <== mon.2 v2:172.21.15.23:3300/0 4 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6dd4004000 con 0x558abcf36570 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 10 client.25556 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 10 client.25556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 20 client.25556 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf46310 0x558abcf48730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 4 client.25556 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:34:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.369+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558abce89ce0 con 0x558abcf46310 2024-03-11T18:34:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.370+0000 7f6de1a00640 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf46310 0x558abcf48730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.370+0000 7f6de1a00640 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf46310 0x558abcf48730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ca8e79d375af844 server_cookie=21a9defba83798a in_seq=0 out_seq=0 2024-03-11T18:34:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.370+0000 7f6dc3400640 10 client.25556 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6dc3400640 1 -- 172.21.15.2:0/563741618 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558abce89ce0 con 0x558abcf46310 2024-03-11T18:34:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6dc3400640 10 client.25556 handle_command_reply: tid=0 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 client.25556 shutdown 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 msgr2=0x558abcf33fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 0x558abcf33fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4011760 tx=0x7f6dd4003c60 comp rx=0 tx=0).stop 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 shutdown_connections 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf46310 0x558abcf48730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf36570 0x558abcf33fe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2dd60 0x558abcf33aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 --2- 172.21.15.2:0/563741618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6dc3400640 0 client.25556 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.371+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 >> 172.21.15.2:0/563741618 conn(0x558abcef44b0 msgr2=0x558abcf29a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.372+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 shutdown_connections 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.372+0000 7f6de71e9740 1 -- 172.21.15.2:0/563741618 wait complete. 2024-03-11T18:34:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.372+0000 7f6de71e9740 20 client.25556 trim_cache size 0 max 16384 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 Processor -- start 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 -- start start 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf307c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcbdbe60 con 0x558abcf2cff0 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf27550 con 0x558abcf2dd60 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.386+0000 7f6de71e9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf39180 con 0x558abcef44b0 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40662/0 (socket says 172.21.15.2:40662) 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 -- 172.21.15.2:0/2320936247 learned_addr learned my addr 172.21.15.2:0/2320936247 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1000640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de2400640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 -- 172.21.15.2:0/2320936247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 msgr2=0x558abcf307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 -- 172.21.15.2:0/2320936247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 msgr2=0x558abcf33e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1a00640 1 -- 172.21.15.2:0/2320936247 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558abce2e530 con 0x558abcef44b0 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de2400640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.387+0000 7f6de1000640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.388+0000 7f6de1a00640 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00966f0 tx=0x7f6dd00965b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d31948c1758d7575 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.388+0000 7f6dce000640 1 -- 172.21.15.2:0/2320936247 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0098330 con 0x558abcef44b0 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.388+0000 7f6dce000640 1 -- 172.21.15.2:0/2320936247 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd0098590 con 0x558abcef44b0 2024-03-11T18:34:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.388+0000 7f6dce000640 1 -- 172.21.15.2:0/2320936247 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000fe50 con 0x558abcef44b0 2024-03-11T18:34:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 -- 172.21.15.2:0/2320936247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 msgr2=0x558abcf35c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00966f0 tx=0x7f6dd00965b0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 -- 172.21.15.2:0/2320936247 shutdown_connections 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcf2dd60 0x558abcf307c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2cff0 0x558abcf33e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 --2- 172.21.15.2:0/2320936247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcef44b0 0x558abcf35c90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 -- 172.21.15.2:0/2320936247 >> 172.21.15.2:0/2320936247 conn(0x558abcf23170 msgr2=0x558abcf31e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 -- 172.21.15.2:0/2320936247 shutdown_connections 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.389+0000 7f6de71e9740 1 -- 172.21.15.2:0/2320936247 wait complete. 2024-03-11T18:34:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 -- start start 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 Processor -- start 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 0x558abcf3c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 0x558abcf3c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf9c820 con 0x558abcf2dd60 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf9c960 con 0x558abcef44b0 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.390+0000 7f6de71e9740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558abcf9cad0 con 0x558abcf2cff0 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de71e9740 10 client.0 _ll_register_callbacks cb 0x7f6dd0009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:41368/0 (socket says 172.21.15.2:41368) 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 -- 172.21.15.2:0/279369833 learned_addr learned my addr 172.21.15.2:0/279369833 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de2400640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 0x558abcf3c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 -- 172.21.15.2:0/279369833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 msgr2=0x558abcf3c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1000640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 0x558abcf3c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 0x558abcf3c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 -- 172.21.15.2:0/279369833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 msgr2=0x558abcf3c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 0x558abcf3c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1a00640 1 -- 172.21.15.2:0/279369833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc1a6910 con 0x558abcef44b0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de1000640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 0x558abcf3c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.391+0000 7f6de2400640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 0x558abcf3c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6de1a00640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000f430 tx=0x7f6dd000b530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c99b9fb4e9dd0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00a2630 con 0x558abcef44b0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558abceb8d30 con 0x558abcef44b0 2024-03-11T18:34:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6dd000fbf0 con 0x558abcef44b0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dd00a2790 con 0x558abcef44b0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6de71e9740 10 client.25559 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558abce542c0 con 0x558abcef44b0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dd00a6020 con 0x558abcef44b0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6dd0023070 con 0x558abcef44b0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.392+0000 7f6de71e9740 10 client.25559 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de71e9740 10 client.25559 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de71e9740 20 client.25559 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf36570 0x558abcf9fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de71e9740 4 client.25559 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558abcb93600 con 0x558abcf36570 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.393+0000 7f6de1000640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf36570 0x558abcf9fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.394+0000 7f6de1000640 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf36570 0x558abcf9fdf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35cf7120b4db5109 server_cookie=12d41bc87b87f66c in_seq=0 out_seq=0 2024-03-11T18:34:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.394+0000 7f6dc3400640 10 client.25559 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6dc3400640 1 -- 172.21.15.2:0/279369833 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558abcb93600 con 0x558abcf36570 2024-03-11T18:34:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6dc3400640 10 client.25559 handle_command_reply: tid=0 2024-03-11T18:34:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 client.25559 shutdown 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 msgr2=0x558abcf3be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000f430 tx=0x7f6dd000b530 comp rx=0 tx=0).stop 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 shutdown_connections 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558abcf36570 0x558abcf9fdf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558abcf2dd60 0x558abcf3c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6dc3400640 0 client.25559 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.401+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558abcf2cff0 0x558abcf3c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.402+0000 7f6de71e9740 1 --2- 172.21.15.2:0/279369833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558abcef44b0 0x558abcf3be50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.402+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 >> 172.21.15.2:0/279369833 conn(0x558abcf23170 msgr2=0x558abcf264b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.402+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 shutdown_connections 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.402+0000 7f6de71e9740 1 -- 172.21.15.2:0/279369833 wait complete. 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.402+0000 7f6de71e9740 20 client.25559 trim_cache size 0 max 16384 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:34:39.405 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 msgr2=0x7f6dc40a4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 0x7f6dc40a4210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6dc802c160 tx=0x7f6dc80046e0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 msgr2=0x7f6ddc073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000d950 tx=0x7f6dd000de10 comp rx=0 tx=0).stop 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 shutdown_connections 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ddc073f00 0x7f6ddc1a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6dc40a1d50 0x7f6dc40a4210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ddc1062a0 0x7f6ddc0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 --2- 172.21.15.2:0/2170494885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ddc1011d0 0x7f6ddc073480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 >> 172.21.15.2:0/2170494885 conn(0x7f6ddc0fca90 msgr2=0x7f6ddc0fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 shutdown_connections 2024-03-11T18:34:39.407 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:39.403+0000 7f6de2e00640 1 -- 172.21.15.2:0/2170494885 wait complete. 2024-03-11T18:34:40.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:40 smithi165 ceph-mon[36634]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:40 smithi023 ceph-mon[36406]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:40 smithi002 ceph-mon[31328]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-11T18:34:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:42 smithi165 ceph-mon[36634]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-11T18:34:42.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:42 smithi023 ceph-mon[36406]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-11T18:34:42.458 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:42 smithi002 ceph-mon[31328]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-11T18:34:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:44 smithi165 ceph-mon[36634]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:44 smithi023 ceph-mon[36406]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:44.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:44 smithi002 ceph-mon[31328]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-11T18:34:46.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:46 smithi165 ceph-mon[36634]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:46.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:46 smithi023 ceph-mon[36406]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:46.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:46 smithi002 ceph-mon[31328]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:48.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:48 smithi002 ceph-mon[31328]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:48.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:48 smithi165 ceph-mon[36634]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:48.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:48 smithi023 ceph-mon[36406]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:50 smithi002 ceph-mon[31328]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-11T18:34:50.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:50 smithi165 ceph-mon[36634]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-11T18:34:50.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:50 smithi023 ceph-mon[36406]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-11T18:34:52.458 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:52 smithi023 ceph-mon[36406]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:52.458 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:52 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:34:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:52 smithi002 ceph-mon[31328]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:52 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 -- 172.21.15.2:0/3413633708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 msgr2=0x7f4888111ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 --2- 172.21.15.2:0/3413633708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f4888111ac0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4874009930 tx=0x7f487402bc30 comp rx=0 tx=0).stop 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 -- 172.21.15.2:0/3413633708 shutdown_connections 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 --2- 172.21.15.2:0/3413633708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4888112000 0x7f48881143f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 --2- 172.21.15.2:0/3413633708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f4888111ac0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 --2- 172.21.15.2:0/3413633708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48880756b0 0x7f4888075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 -- 172.21.15.2:0/3413633708 >> 172.21.15.2:0/3413633708 conn(0x7f48880fee70 msgr2=0x7f4888101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 -- 172.21.15.2:0/3413633708 shutdown_connections 2024-03-11T18:34:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.584+0000 7f488f800640 1 -- 172.21.15.2:0/3413633708 wait complete. 2024-03-11T18:34:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.585+0000 7f488f800640 1 Processor -- start 2024-03-11T18:34:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 -- start start 2024-03-11T18:34:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f488819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 0x7f48881a2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48881a2d30 con 0x7f48880756b0 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48881a2ea0 con 0x7f4888075fd0 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488f800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48881a3060 con 0x7f4888112000 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.586+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39562/0 (socket says 172.21.15.2:39562) 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488ce00640 1 -- 172.21.15.2:0/2820368164 learned_addr learned my addr 172.21.15.2:0/2820368164 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f4887e00640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f488819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 0x7f48881a2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 -- 172.21.15.2:0/2820368164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 msgr2=0x7f488819e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f488819e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 -- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 msgr2=0x7f488819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488d800640 1 -- 172.21.15.2:0/2820368164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874009560 con 0x7f4888112000 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.587+0000 7f488ce00640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f488d800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 0x7f48881a2730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4880005fd0 tx=0x7f488000edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f4885600640 1 -- 172.21.15.2:0/2820368164 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880004d20 con 0x7f4888112000 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f4885600640 1 -- 172.21.15.2:0/2820368164 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4880004e80 con 0x7f4888112000 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48881a3340 con 0x7f4888112000 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f4885600640 1 -- 172.21.15.2:0/2820368164 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800116d0 con 0x7f4888112000 2024-03-11T18:34:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.588+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48881aabc0 con 0x7f4888112000 2024-03-11T18:34:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.591+0000 7f48925dd740 1 Processor -- start 2024-03-11T18:34:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.593+0000 7f4885600640 1 -- 172.21.15.2:0/2820368164 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 26) v1 ==== 136511+0+0 (secure 0 0 0) 0x7f4880005990 con 0x7f4888112000 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.593+0000 7f48925dd740 1 -- start start 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f62720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2f12620 con 0x55abb2f56db0 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2ec48c0 con 0x55abb2f57b20 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.594+0000 7f48925dd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2ec1df0 con 0x55abb2f60330 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f4887e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39576/0 (socket says 172.21.15.2:39576) 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f4887e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39070/0 (socket says 172.21.15.2:39070) 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 -- 172.21.15.2:0/2585326140 learned_addr learned my addr 172.21.15.2:0/2585326140 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f4887e00640 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f62720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 -- 172.21.15.2:0/2585326140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 msgr2=0x55abb2f62720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f62720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 -- 172.21.15.2:0/2585326140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 msgr2=0x55abb2f56870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f488ce00640 1 -- 172.21.15.2:0/2585326140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abb2e3eb00 con 0x55abb2f56db0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f4887e00640 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.595+0000 7f4885600640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 0x7f48700a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.596+0000 7f4885600640 1 -- 172.21.15.2:0/2820368164 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f48800cdee0 con 0x7f4888112000 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.596+0000 7f4887e00640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 0x7f48700a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.596+0000 7f488ce00640 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f487c011650 tx=0x7f487c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72319327e0a3bef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.596+0000 7f487a000640 1 -- 172.21.15.2:0/2585326140 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00f6c0 con 0x55abb2f56db0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.597+0000 7f487a000640 1 -- 172.21.15.2:0/2585326140 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f487c00fd00 con 0x55abb2f56db0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.597+0000 7f487a000640 1 -- 172.21.15.2:0/2585326140 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c019e20 con 0x55abb2f56db0 2024-03-11T18:34:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.597+0000 7f4887e00640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 0x7f48700a41f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4874009900 tx=0x7f4874005b50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.598+0000 7f48925dd740 1 -- 172.21.15.2:0/2585326140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 msgr2=0x55abb2f57230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.598+0000 7f48925dd740 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f487c011650 tx=0x7f487c009f90 comp rx=0 tx=0).stop 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 -- 172.21.15.2:0/2585326140 shutdown_connections 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f62720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f57230 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 --2- 172.21.15.2:0/2585326140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f57b20 0x55abb2f56870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 -- 172.21.15.2:0/2585326140 >> 172.21.15.2:0/2585326140 conn(0x55abb2f1e4b0 msgr2=0x55abb2f51aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 -- 172.21.15.2:0/2585326140 shutdown_connections 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 -- 172.21.15.2:0/2585326140 wait complete. 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.599+0000 7f48925dd740 1 -- start start 2024-03-11T18:34:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.600+0000 7f48925dd740 1 Processor -- start 2024-03-11T18:34:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f57b20 0x55abb2f5d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f5ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39072/0 (socket says 172.21.15.2:39072) 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2ebd5d0 con 0x55abb2f57b20 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 -- 172.21.15.2:0/1465191669 learned_addr learned my addr 172.21.15.2:0/1465191669 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2e62b50 con 0x55abb2f56db0 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2e627f0 con 0x55abb2f60330 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f48925dd740 10 client.0 _ll_register_callbacks cb 0x55abb2e76550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f4887e00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f57b20 0x55abb2f5d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488d800640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f5ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 -- 172.21.15.2:0/1465191669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 msgr2=0x55abb2f5ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f5ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 -- 172.21.15.2:0/1465191669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f57b20 msgr2=0x55abb2f5d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f57b20 0x55abb2f5d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.601+0000 7f488ce00640 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c009e00 con 0x55abb2f56db0 2024-03-11T18:34:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.602+0000 7f488ce00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f487c0097f0 tx=0x7f487c01ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ac57f516d420a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.602+0000 7f486ea00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:34:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.602+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c004240 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.602+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f487c0043a0 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.603+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55abb2e8fcd0 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.603+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f487c005040 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.603+0000 7f48925dd740 10 client.25571 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.603+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55abb2ebbc50 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.603+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55abb2e8fcd0 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f487c0046c0 con 0x55abb2f56db0 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 10 client.25571 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 10 client.25571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 20 client.25571 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2f70250 0x55abb2f72680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 4 client.25571 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:34:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abb2eb3ce0 con 0x55abb2f70250 2024-03-11T18:34:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.604+0000 7f4887e00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2f70250 0x55abb2f72680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.605+0000 7f4887e00640 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2f70250 0x55abb2f72680 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2432cb24f4a35020 server_cookie=435fb2e27234a708 in_seq=0 out_seq=0 2024-03-11T18:34:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.605+0000 7f486ea00640 10 client.25571 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f486ea00640 1 -- 172.21.15.2:0/1465191669 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55abb2eb3ce0 con 0x55abb2f70250 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f486ea00640 10 client.25571 handle_command_reply: tid=0 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 client.25571 shutdown 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 msgr2=0x55abb2f5d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f487c0097f0 tx=0x7f487c01ada0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 shutdown_connections 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2f70250 0x55abb2f72680 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f486ea00640 0 client.25571 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f60330 0x55abb2f5ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f57b20 0x55abb2f5d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 --2- 172.21.15.2:0/1465191669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f56db0 0x55abb2f5d370 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.606+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 >> 172.21.15.2:0/1465191669 conn(0x55abb2f1e4b0 msgr2=0x55abb2f537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.607+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 shutdown_connections 2024-03-11T18:34:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.607+0000 7f48925dd740 1 -- 172.21.15.2:0/1465191669 wait complete. 2024-03-11T18:34:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.607+0000 7f48925dd740 20 client.25571 trim_cache size 0 max 16384 2024-03-11T18:34:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.635+0000 7f48925dd740 1 Processor -- start 2024-03-11T18:34:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.635+0000 7f48925dd740 1 -- start start 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f5faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f67150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2c05e60 con 0x55abb2f56db0 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2f6c170 con 0x55abb2f60330 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f48925dd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2eb3ce0 con 0x55abb2f57b20 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f488d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f488d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39080/0 (socket says 172.21.15.2:39080) 2024-03-11T18:34:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.636+0000 7f488d800640 1 -- 172.21.15.2:0/3432949216 learned_addr learned my addr 172.21.15.2:0/3432949216 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 -- 172.21.15.2:0/3432949216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 msgr2=0x55abb2f67150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f67150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 -- 172.21.15.2:0/3432949216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 msgr2=0x55abb2f5faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f5faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 -- 172.21.15.2:0/3432949216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abb2e58530 con 0x55abb2f60330 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f488d800640 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4880096920 tx=0x7f4880099ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6076161dcf752837 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f4879600640 1 -- 172.21.15.2:0/3432949216 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488009e610 con 0x55abb2f60330 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f4879600640 1 -- 172.21.15.2:0/3432949216 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48800a2c60 con 0x55abb2f60330 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.637+0000 7f4879600640 1 -- 172.21.15.2:0/3432949216 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800052b0 con 0x55abb2f60330 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.638+0000 7f48925dd740 1 -- 172.21.15.2:0/3432949216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 msgr2=0x55abb2f67690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.638+0000 7f48925dd740 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4880096920 tx=0x7f4880099ee0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.638+0000 7f48925dd740 1 -- 172.21.15.2:0/3432949216 shutdown_connections 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.638+0000 7f48925dd740 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f67690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f67150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 --2- 172.21.15.2:0/3432949216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f5faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 -- 172.21.15.2:0/3432949216 >> 172.21.15.2:0/3432949216 conn(0x55abb2f1e4b0 msgr2=0x55abb2f77020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 -- 172.21.15.2:0/3432949216 shutdown_connections 2024-03-11T18:34:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 -- 172.21.15.2:0/3432949216 wait complete. 2024-03-11T18:34:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.639+0000 7f48925dd740 1 -- start start 2024-03-11T18:34:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 Processor -- start 2024-03-11T18:34:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f67690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f63a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f641a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2f64870 con 0x55abb2f56db0 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2fc67a0 con 0x55abb2f60330 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abb2fc68e0 con 0x55abb2f57b20 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.640+0000 7f48925dd740 10 client.0 _ll_register_callbacks cb 0x7f487c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f67690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f488d800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f63a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f488ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f67690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39602/0 (socket says 172.21.15.2:39602) 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f488ce00640 1 -- 172.21.15.2:0/2888755416 learned_addr learned my addr 172.21.15.2:0/2888755416 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:34:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 -- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 msgr2=0x55abb2f641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 -- 172.21.15.2:0/2888755416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 msgr2=0x55abb2f67690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:34:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f67690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 -- 172.21.15.2:0/2888755416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48881aabc0 con 0x55abb2f57b20 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f4887e00640 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f63a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4874036040 tx=0x7f48740048d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=491acbe25814e6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.641+0000 7f486ea00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874053370 con 0x55abb2f57b20 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55abb2ee2d30 con 0x55abb2f57b20 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f487404a040 con 0x55abb2f57b20 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48740534d0 con 0x55abb2f57b20 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f48925dd740 10 client.25582 fetch_fsmap learned FSMap version 25 2024-03-11T18:34:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.642+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55abb2e7e2c0 con 0x55abb2f57b20 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55abb2ee2d30 con 0x55abb2f57b20 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 25) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f487403c040 con 0x55abb2f57b20 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 10 client.25582 fetch_fsmap finished waiting for FSMap version 25 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 10 client.25582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 20 client.25582 populate_metadata read hostname 'smithi002' 2024-03-11T18:34:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2fc80b0 0x55abb2fca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:34:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 4 client.25582 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:34:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55abb2bbd600 con 0x55abb2fc80b0 2024-03-11T18:34:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.644+0000 7f488ce00640 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2fc80b0 0x55abb2fca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:34:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.645+0000 7f488ce00640 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2fc80b0 0x55abb2fca4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7637a080a33a78b server_cookie=fc34473427904783 in_seq=0 out_seq=0 2024-03-11T18:34:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.645+0000 7f486ea00640 10 client.25582 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.649+0000 7f486ea00640 1 -- 172.21.15.2:0/2888755416 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55abb2bbd600 con 0x55abb2fc80b0 2024-03-11T18:34:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.649+0000 7f486ea00640 10 client.25582 handle_command_reply: tid=0 2024-03-11T18:34:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.649+0000 7f48925dd740 1 client.25582 shutdown 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.649+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 msgr2=0x55abb2f63a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f63a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4874036040 tx=0x7f48740048d0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 shutdown_connections 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55abb2fc80b0 0x55abb2fca4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abb2f60330 0x55abb2f641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55abb2f57b20 0x55abb2f63a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 --2- 172.21.15.2:0/2888755416 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55abb2f56db0 0x55abb2f67690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 >> 172.21.15.2:0/2888755416 conn(0x55abb2f1e4b0 msgr2=0x55abb2f5bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f486ea00640 0 client.25582 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 shutdown_connections 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 1 -- 172.21.15.2:0/2888755416 wait complete. 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.650+0000 7f48925dd740 20 client.25582 trim_cache size 0 max 16384 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:34:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 msgr2=0x7f48700a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 0x7f48700a41f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4874009900 tx=0x7f4874005b50 comp rx=0 tx=0).stop 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 msgr2=0x7f48881a2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 0x7f48881a2730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4880005fd0 tx=0x7f488000edd0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 shutdown_connections 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4888112000 0x7f48881a2730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4888075fd0 0x7f488819e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f48700a1d30 0x7f48700a41f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 --2- 172.21.15.2:0/2820368164 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f48880756b0 0x7f488819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:34:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.651+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 >> 172.21.15.2:0/2820368164 conn(0x7f48880fee70 msgr2=0x7f4888113720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:34:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.652+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 shutdown_connections 2024-03-11T18:34:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:34:52.652+0000 7f488f800640 1 -- 172.21.15.2:0/2820368164 wait complete. 2024-03-11T18:34:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:52 smithi165 ceph-mon[36634]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-11T18:34:52.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:52 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:34:53.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:53 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:34:53.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:53 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:34:53.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:53 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:34:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:53 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:34:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:53 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:34:53.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:53 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:34:53.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:53 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:34:53.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:53 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:34:53.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:53 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:34:54.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:54 smithi002 ceph-mon[31328]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:54.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:54 smithi023 ceph-mon[36406]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:54.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:54 smithi165 ceph-mon[36634]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-11T18:34:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:56 smithi002 ceph-mon[31328]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-11T18:34:56.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:56 smithi165 ceph-mon[36634]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-11T18:34:56.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:56 smithi023 ceph-mon[36406]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-11T18:34:58.329 INFO:tasks.workunit.client.0.smithi002.stderr:Updating files: 62% (7731/12344) Updating files: 63% (7777/12344) Updating files: 64% (7901/12344) Updating files: 65% (8024/12344) Updating files: 66% (8148/12344) Updating files: 67% (8271/12344) Updating files: 68% (8394/12344) Updating files: 69% (8518/12344) Updating files: 70% (8641/12344) Updating files: 71% (8765/12344) Updating files: 72% (8888/12344) Updating files: 73% (9012/12344) Updating files: 74% (9135/12344) Updating files: 75% (9258/12344) Updating files: 76% (9382/12344) Updating files: 77% (9505/12344) Updating files: 78% (9629/12344) Updating files: 79% (9752/12344) Updating files: 80% (9876/12344) Updating files: 81% (9999/12344) Updating files: 82% (10123/12344) Updating files: 83% (10246/12344) Updating files: 84% (10369/12344) Updating files: 85% (10493/12344) Updating files: 86% (10616/12344) Updating files: 87% (10740/12344) Updating files: 88% (10863/12344) Updating files: 89% (10987/12344) Updating files: 90% (11110/12344) Updating files: 91% (11234/12344) Updating files: 92% (11357/12344) Updating files: 93% (11480/12344) Updating files: 94% (11604/12344) Updating files: 95% (11727/12344) Updating files: 96% (11851/12344) Updating files: 97% (11974/12344) Updating files: 98% (12098/12344) Updating files: 99% (12221/12344) Updating files: 100% (12344/12344) Updating files: 100% (12344/12344), done. 2024-03-11T18:34:58.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:34:58 smithi002 ceph-mon[31328]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:34:58.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:34:58 smithi165 ceph-mon[36634]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:34:58.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:34:58 smithi023 ceph-mon[36406]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:34:58.748 INFO:tasks.workunit.client.0.smithi002.stderr:Note: switching to 'b6a4997be2b8bb74411268cc42753f29ae9a78ad'. 2024-03-11T18:34:58.749 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.749 INFO:tasks.workunit.client.0.smithi002.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T18:34:58.749 INFO:tasks.workunit.client.0.smithi002.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T18:34:58.749 INFO:tasks.workunit.client.0.smithi002.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T18:34:58.749 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr: git switch -c 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr:Or undo this operation with: 2024-03-11T18:34:58.750 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.751 INFO:tasks.workunit.client.0.smithi002.stderr: git switch - 2024-03-11T18:34:58.751 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.751 INFO:tasks.workunit.client.0.smithi002.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T18:34:58.751 INFO:tasks.workunit.client.0.smithi002.stderr: 2024-03-11T18:34:58.751 INFO:tasks.workunit.client.0.smithi002.stderr:HEAD is now at b6a4997be2b Merge pull request #56060 from rhcs-dashboard/wip-64808-squid 2024-03-11T18:34:58.756 DEBUG:teuthology.orchestra.run.smithi002:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T18:34:58.855 INFO:tasks.workunit.client.0.smithi002.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T18:34:58.862 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T18:34:58.862 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T18:34:59.783 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T18:34:59.922 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T18:35:00.015 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T18:35:00.020 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T18:35:00.021 INFO:tasks.workunit.client.0.smithi002.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T18:35:00.114 INFO:tasks.workunit.client.0.smithi002.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T18:35:00.124 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:35:00.124 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T18:35:00.176 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-11T18:35:00.178 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-11T18:35:00.178 DEBUG:teuthology.orchestra.run.smithi002:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-11T18:35:00.256 INFO:tasks.workunit.client.0.smithi002.stderr:+ dbench 1 2024-03-11T18:35:00.265 INFO:tasks.workunit.client.0.smithi002.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-11T18:35:00.265 INFO:tasks.workunit.client.0.smithi002.stdout: 2024-03-11T18:35:00.265 INFO:tasks.workunit.client.0.smithi002.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-11T18:35:00.274 INFO:tasks.workunit.client.0.smithi002.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-11T18:35:00.374 INFO:tasks.workunit.client.0.smithi002.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-11T18:35:00.374 INFO:tasks.workunit.client.0.smithi002.stdout:releasing clients 2024-03-11T18:35:00.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:00 smithi002 ceph-mon[31328]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:00.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:00 smithi165 ceph-mon[36634]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:00 smithi023 ceph-mon[36406]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:01.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 1909 26.45 MB/sec warmup 1 sec latency 12.442 ms 2024-03-11T18:35:02.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 4129 20.04 MB/sec warmup 2 sec latency 11.534 ms 2024-03-11T18:35:02.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:02 smithi002 ceph-mon[31328]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:02.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:02 smithi023 ceph-mon[36406]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:02.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:02 smithi165 ceph-mon[36634]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-11T18:35:03.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 6335 17.03 MB/sec warmup 3 sec latency 15.727 ms 2024-03-11T18:35:03.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:03 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:35:03.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:03 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:35:03.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:03 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:35:04.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 7914 14.93 MB/sec warmup 4 sec latency 14.057 ms 2024-03-11T18:35:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:04 smithi002 ceph-mon[31328]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:35:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:04 smithi165 ceph-mon[36634]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:35:04.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:04 smithi023 ceph-mon[36406]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T18:35:05.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 10654 15.33 MB/sec warmup 5 sec latency 11.708 ms 2024-03-11T18:35:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 -- 172.21.15.2:0/1553137951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 msgr2=0x7f525010d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 --2- 172.21.15.2:0/1553137951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525010d260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5238009900 tx=0x7f523802bcc0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 -- 172.21.15.2:0/1553137951 shutdown_connections 2024-03-11T18:35:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 --2- 172.21.15.2:0/1553137951 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 0x7f525010fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 --2- 172.21.15.2:0/1553137951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525010d260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 --2- 172.21.15.2:0/1553137951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 0x7f5250073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 -- 172.21.15.2:0/1553137951 >> 172.21.15.2:0/1553137951 conn(0x7f52500fca30 msgr2=0x7f52500fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 -- 172.21.15.2:0/1553137951 shutdown_connections 2024-03-11T18:35:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.798+0000 7f5255000640 1 -- 172.21.15.2:0/1553137951 wait complete. 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.799+0000 7f5255000640 1 Processor -- start 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 -- start start 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 0x7f525019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 0x7f5250199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250199c90 con 0x7f52500735b0 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250199e00 con 0x7f5250073ed0 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f5255000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5250199fc0 con 0x7f525010d7a0 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f524ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.800+0000 7f524f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 0x7f525019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52690/0 (socket says 172.21.15.2:52690) 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 0x7f5250199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 -- 172.21.15.2:0/1599652804 learned_addr learned my addr 172.21.15.2:0/1599652804 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 -- 172.21.15.2:0/1599652804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 msgr2=0x7f5250199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 0x7f5250199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 -- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 msgr2=0x7f525019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 0x7f525019f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 -- 172.21.15.2:0/1599652804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5238009560 con 0x7f5250073ed0 2024-03-11T18:35:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.801+0000 7f524ea00640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5238000c00 tx=0x7f523802c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.802+0000 7f5247e00640 1 -- 172.21.15.2:0/1599652804 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523802dc50 con 0x7f5250073ed0 2024-03-11T18:35:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.802+0000 7f5247e00640 1 -- 172.21.15.2:0/1599652804 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f523802ddb0 con 0x7f5250073ed0 2024-03-11T18:35:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.802+0000 7f5247e00640 1 -- 172.21.15.2:0/1599652804 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238003b80 con 0x7f5250073ed0 2024-03-11T18:35:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.802+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525019a240 con 0x7f5250073ed0 2024-03-11T18:35:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.802+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52501a64e0 con 0x7f5250073ed0 2024-03-11T18:35:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.805+0000 7f525926f740 1 Processor -- start 2024-03-11T18:35:05.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.805+0000 7f525926f740 1 -- start start 2024-03-11T18:35:05.808 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.805+0000 7f5247e00640 1 -- 172.21.15.2:0/1599652804 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136511+0+0 (secure 0 0 0) 0x7f52380054b0 con 0x7f5250073ed0 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98b69620 con 0x557b98bb7570 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98b1b8c0 con 0x557b98badff0 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f525926f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98b18df0 con 0x557b98baed60 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41036/0 (socket says 172.21.15.2:41036) 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52698/0 (socket says 172.21.15.2:52698) 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 -- 172.21.15.2:0/460296337 learned_addr learned my addr 172.21.15.2:0/460296337 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 -- 172.21.15.2:0/460296337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 msgr2=0x557b98bae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bae470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 -- 172.21.15.2:0/460296337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 msgr2=0x557b98bb9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.806+0000 7f524fe00640 1 -- 172.21.15.2:0/460296337 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b98a95b00 con 0x557b98baed60 2024-03-11T18:35:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f524fe00640 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5248009f70 tx=0x7f5248010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455d73385efa43ef server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f5247e00640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 0x7f52340a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f5244c00640 1 -- 172.21.15.2:0/460296337 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5248004d20 con 0x557b98baed60 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f5244c00640 1 -- 172.21.15.2:0/460296337 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5248004e80 con 0x557b98baed60 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f5244c00640 1 -- 172.21.15.2:0/460296337 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480129b0 con 0x557b98baed60 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.807+0000 7f5247e00640 1 -- 172.21.15.2:0/1599652804 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f52380f02c0 con 0x7f5250073ed0 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f524f400640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 0x7f52340a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 -- 172.21.15.2:0/460296337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 msgr2=0x557b98badab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5248009f70 tx=0x7f5248010800 comp rx=0 tx=0).stop 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 -- 172.21.15.2:0/460296337 shutdown_connections 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 --2- 172.21.15.2:0/460296337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98badab0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 -- 172.21.15.2:0/460296337 >> 172.21.15.2:0/460296337 conn(0x557b98b754b0 msgr2=0x557b98ba8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f524f400640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 0x7f52340a41e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x557b98baf4c0 tx=0x7f5240005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 -- 172.21.15.2:0/460296337 shutdown_connections 2024-03-11T18:35:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.808+0000 7f525926f740 1 -- 172.21.15.2:0/460296337 wait complete. 2024-03-11T18:35:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.809+0000 7f525926f740 1 -- start start 2024-03-11T18:35:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.809+0000 7f525926f740 1 Processor -- start 2024-03-11T18:35:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.809+0000 7f525926f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 0x557b98bb4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.809+0000 7f525926f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.809+0000 7f525926f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f525926f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98b145d0 con 0x557b98bb7570 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f525926f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98ab9b50 con 0x557b98baed60 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f525926f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98ab97f0 con 0x557b98badff0 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f525926f740 10 client.0 _ll_register_callbacks cb 0x557b98acd550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52706/0 (socket says 172.21.15.2:52706) 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 -- 172.21.15.2:0/2838251001 learned_addr learned my addr 172.21.15.2:0/2838251001 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524f400640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 0x557b98bb4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 -- 172.21.15.2:0/2838251001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 msgr2=0x557b98bb4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 0x557b98bb4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 -- 172.21.15.2:0/2838251001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 msgr2=0x557b98bb5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 -- 172.21.15.2:0/2838251001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5248009bd0 con 0x557b98baed60 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524f400640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 0x557b98bb4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f524ea00640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5238005ec0 tx=0x7f52380035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6878b66ef8413c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.810+0000 7f523d600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380bc8e0 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f52380bde40 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b98ae6cd0 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52380bc650 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 10 client.25589 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557b98b12c50 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52380d4070 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f523803c070 con 0x557b98baed60 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 10 client.25589 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 20 client.25589 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98bc7450 0x557b98bc9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 4 client.25589 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.811+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b98b0ace0 con 0x557b98bc7450 2024-03-11T18:35:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.812+0000 7f524f400640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98bc7450 0x557b98bc9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.812+0000 7f524f400640 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98bc7450 0x557b98bc9870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311736da11c81eab server_cookie=513274e3ca67cfcb in_seq=0 out_seq=0 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.812+0000 7f523d600640 10 client.25589 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f523d600640 1 -- 172.21.15.2:0/2838251001 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557b98b0ace0 con 0x557b98bc7450 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f523d600640 10 client.25589 handle_command_reply: tid=0 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 client.25589 shutdown 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 msgr2=0x557b98bb4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5238005ec0 tx=0x7f52380035f0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 shutdown_connections 2024-03-11T18:35:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98bc7450 0x557b98bc9870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98bb7570 0x557b98bb5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98baed60 0x557b98bb4b50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f523d600640 0 client.25589 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 --2- 172.21.15.2:0/2838251001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98badff0 0x557b98bb4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 >> 172.21.15.2:0/2838251001 conn(0x557b98b754b0 msgr2=0x557b98baaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.814+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 shutdown_connections 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.815+0000 7f525926f740 1 -- 172.21.15.2:0/2838251001 wait complete. 2024-03-11T18:35:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.815+0000 7f525926f740 20 client.25589 trim_cache size 0 max 16384 2024-03-11T18:35:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.829+0000 7f525926f740 1 Processor -- start 2024-03-11T18:35:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.829+0000 7f525926f740 1 -- start start 2024-03-11T18:35:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.829+0000 7f525926f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f525926f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 0x557b98bb4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f525926f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 0x557b98bb18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f525926f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9885ce60 con 0x557b98baed60 2024-03-11T18:35:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f525926f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98baf350 con 0x557b98badff0 2024-03-11T18:35:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f525926f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98ba3230 con 0x557b98bb7570 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52720/0 (socket says 172.21.15.2:52720) 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524f400640 1 -- 172.21.15.2:0/1748405196 learned_addr learned my addr 172.21.15.2:0/1748405196 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 0x557b98bb4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524fe00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 0x557b98bb18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 -- 172.21.15.2:0/1748405196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 msgr2=0x557b98bb18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 0x557b98bb18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 -- 172.21.15.2:0/1748405196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 msgr2=0x557b98bb6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524ea00640 1 -- 172.21.15.2:0/1748405196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b98aaf530 con 0x557b98baed60 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524f400640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.830+0000 7f524fe00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 0x557b98bb18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.831+0000 7f524ea00640 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 0x557b98bb4eb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f52380bc810 tx=0x7f5238003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1342fc8ed8c6e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.831+0000 7f523fe00640 1 -- 172.21.15.2:0/1748405196 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380bc650 con 0x557b98baed60 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.831+0000 7f523fe00640 1 -- 172.21.15.2:0/1748405196 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52380d4070 con 0x557b98baed60 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.831+0000 7f523fe00640 1 -- 172.21.15.2:0/1748405196 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380d0dc0 con 0x557b98baed60 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- 172.21.15.2:0/1748405196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 msgr2=0x557b98bb4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 0x557b98bb4eb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f52380bc810 tx=0x7f5238003680 comp rx=0 tx=0).stop 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- 172.21.15.2:0/1748405196 shutdown_connections 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98bb7570 0x557b98bb18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98baed60 0x557b98bb4eb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 --2- 172.21.15.2:0/1748405196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98badff0 0x557b98bb6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- 172.21.15.2:0/1748405196 >> 172.21.15.2:0/1748405196 conn(0x557b98b754b0 msgr2=0x557b98bb2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- 172.21.15.2:0/1748405196 shutdown_connections 2024-03-11T18:35:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- 172.21.15.2:0/1748405196 wait complete. 2024-03-11T18:35:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.832+0000 7f525926f740 1 -- start start 2024-03-11T18:35:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 Processor -- start 2024-03-11T18:35:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98badff0 0x557b98bbe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98bb7570 0x557b98bbd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98bbd8e0 con 0x557b98badff0 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98bbda50 con 0x557b98bb7570 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b98c1d970 con 0x557b98baed60 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f525926f740 10 client.0 _ll_register_callbacks cb 0x7f523802df20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f524f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98badff0 0x557b98bbe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f524ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f524fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98bb7570 0x557b98bbd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f524ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41060/0 (socket says 172.21.15.2:41060) 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.833+0000 7f524ea00640 1 -- 172.21.15.2:0/251192007 learned_addr learned my addr 172.21.15.2:0/251192007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 -- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98bb7570 msgr2=0x557b98bbd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98bb7570 0x557b98bbd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 -- 172.21.15.2:0/251192007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98badff0 msgr2=0x557b98bbe7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98badff0 0x557b98bbe7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 -- 172.21.15.2:0/251192007 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5240009e50 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f524ea00640 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5238009320 tx=0x7f52380d0ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1314d1f0ab73163e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f523d600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52380bc8e0 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f52380bde40 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.834+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b98b39d30 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52380c4f10 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 10 client.25600 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557b98b12c50 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557b98b39d30 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f523803c070 con 0x557b98baed60 2024-03-11T18:35:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 10 client.25600 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 10 client.25600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 20 client.25600 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98c1ecc0 0x557b98c210e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 4 client.25600 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557b98814600 con 0x557b98c1ecc0 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.835+0000 7f524f400640 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98c1ecc0 0x557b98c210e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.836+0000 7f524f400640 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98c1ecc0 0x557b98c210e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=215f6d5d7a8d1b0f server_cookie=813fa6b0eeb6dde in_seq=0 out_seq=0 2024-03-11T18:35:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:05.836+0000 7f523d600640 10 client.25600 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.041+0000 7f523d600640 1 -- 172.21.15.2:0/251192007 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557b98814600 con 0x557b98c1ecc0 2024-03-11T18:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.041+0000 7f523d600640 10 client.25600 handle_command_reply: tid=0 2024-03-11T18:35:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.041+0000 7f525926f740 1 client.25600 shutdown 2024-03-11T18:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 msgr2=0x557b98bbcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5238009320 tx=0x7f52380d0ef0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 shutdown_connections 2024-03-11T18:35:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557b98c1ecc0 0x557b98c210e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557b98bb7570 0x557b98bbd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557b98baed60 0x557b98bbcb90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 --2- 172.21.15.2:0/251192007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557b98badff0 0x557b98bbe7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 >> 172.21.15.2:0/251192007 conn(0x557b98b754b0 msgr2=0x557b98ba74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.042+0000 7f523d600640 0 client.25600 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.043+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 shutdown_connections 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.043+0000 7f525926f740 1 -- 172.21.15.2:0/251192007 wait complete. 2024-03-11T18:35:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.043+0000 7f525926f740 20 client.25600 trim_cache size 0 max 16384 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 msgr2=0x7f52340a41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 0x7f52340a41e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x557b98baf4c0 tx=0x7f5240005f90 comp rx=0 tx=0).stop 2024-03-11T18:35:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 msgr2=0x7f525019f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5238000c00 tx=0x7f523802c730 comp rx=0 tx=0).stop 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 shutdown_connections 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f525010d7a0 0x7f5250199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5250073ed0 0x7f525019f980 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f52340a1d20 0x7f52340a41e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 --2- 172.21.15.2:0/1599652804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52500735b0 0x7f525019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 >> 172.21.15.2:0/1599652804 conn(0x7f52500fca30 msgr2=0x7f52500fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 shutdown_connections 2024-03-11T18:35:06.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:06.044+0000 7f5255000640 1 -- 172.21.15.2:0/1599652804 wait complete. 2024-03-11T18:35:06.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 11836 13.39 MB/sec warmup 6 sec latency 66.061 ms 2024-03-11T18:35:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:06 smithi002 ceph-mon[31328]: pgmap v44: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 352 KiB/s wr, 62 op/s 2024-03-11T18:35:06.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:06 smithi023 ceph-mon[36406]: pgmap v44: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 352 KiB/s wr, 62 op/s 2024-03-11T18:35:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:06 smithi165 ceph-mon[36634]: pgmap v44: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 352 KiB/s wr, 62 op/s 2024-03-11T18:35:07.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 14507 13.87 MB/sec warmup 7 sec latency 9.225 ms 2024-03-11T18:35:08.375 INFO:tasks.workunit.client.0.smithi002.stdout: 1 16069 12.85 MB/sec warmup 8 sec latency 25.103 ms 2024-03-11T18:35:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:08 smithi002 ceph-mon[31328]: pgmap v45: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 352 KiB/s wr, 59 op/s 2024-03-11T18:35:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:08 smithi023 ceph-mon[36406]: pgmap v45: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 352 KiB/s wr, 59 op/s 2024-03-11T18:35:08.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:08 smithi165 ceph-mon[36634]: pgmap v45: 129 pgs: 129 active+clean; 4.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 352 KiB/s wr, 59 op/s 2024-03-11T18:35:09.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 18265 12.95 MB/sec warmup 9 sec latency 9.354 ms 2024-03-11T18:35:10.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 20354 12.46 MB/sec warmup 10 sec latency 13.561 ms 2024-03-11T18:35:10.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:10 smithi002 ceph-mon[31328]: pgmap v46: 129 pgs: 129 active+clean; 9.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 841 KiB/s wr, 224 op/s 2024-03-11T18:35:10.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:10 smithi023 ceph-mon[36406]: pgmap v46: 129 pgs: 129 active+clean; 9.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 841 KiB/s wr, 224 op/s 2024-03-11T18:35:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:10 smithi165 ceph-mon[36634]: pgmap v46: 129 pgs: 129 active+clean; 9.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 841 KiB/s wr, 224 op/s 2024-03-11T18:35:11.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 22062 12.37 MB/sec warmup 11 sec latency 17.285 ms 2024-03-11T18:35:12.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 24312 12.36 MB/sec warmup 12 sec latency 35.162 ms 2024-03-11T18:35:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:12 smithi002 ceph-mon[31328]: pgmap v47: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:12.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:12 smithi023 ceph-mon[36406]: pgmap v47: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:12.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:12 smithi165 ceph-mon[36634]: pgmap v47: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:13.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 25867 11.98 MB/sec warmup 13 sec latency 11.833 ms 2024-03-11T18:35:13.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:13 smithi002 ceph-mon[31328]: pgmap v48: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:13.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:13 smithi165 ceph-mon[36634]: pgmap v48: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:13.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:13 smithi023 ceph-mon[36406]: pgmap v48: 129 pgs: 129 active+clean; 9.5 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 862 KiB/s wr, 232 op/s 2024-03-11T18:35:14.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 28647 12.40 MB/sec warmup 14 sec latency 8.909 ms 2024-03-11T18:35:15.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 29947 11.94 MB/sec warmup 15 sec latency 20.991 ms 2024-03-11T18:35:15.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:15 smithi002 ceph-mon[31328]: pgmap v49: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 300 op/s 2024-03-11T18:35:15.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:15 smithi002 ceph-mon[31328]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-11T18:35:15.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:15 smithi165 ceph-mon[36634]: pgmap v49: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 300 op/s 2024-03-11T18:35:15.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:15 smithi165 ceph-mon[36634]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-11T18:35:15.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:15 smithi023 ceph-mon[36406]: pgmap v49: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 300 op/s 2024-03-11T18:35:15.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:15 smithi023 ceph-mon[36406]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-11T18:35:16.376 INFO:tasks.workunit.client.0.smithi002.stdout: 1 32353 12.07 MB/sec warmup 16 sec latency 12.432 ms 2024-03-11T18:35:17.377 INFO:tasks.workunit.client.0.smithi002.stdout: 1 34373 11.77 MB/sec warmup 17 sec latency 13.280 ms 2024-03-11T18:35:17.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:17 smithi002 ceph-mon[31328]: pgmap v50: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 1019 KiB/s wr, 244 op/s 2024-03-11T18:35:17.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:17 smithi165 ceph-mon[36634]: pgmap v50: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 1019 KiB/s wr, 244 op/s 2024-03-11T18:35:17.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:17 smithi023 ceph-mon[36406]: pgmap v50: 129 pgs: 129 active+clean; 14 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 1019 KiB/s wr, 244 op/s 2024-03-11T18:35:18.377 INFO:tasks.workunit.client.0.smithi002.stdout: 1 36240 11.81 MB/sec warmup 18 sec latency 13.696 ms 2024-03-11T18:35:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 -- 172.21.15.2:0/64797995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94103410 msgr2=0x7f1e941037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 --2- 172.21.15.2:0/64797995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94103410 0x7f1e941037f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e84009930 tx=0x7f1e8402bc20 comp rx=0 tx=0).stop 2024-03-11T18:35:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 -- 172.21.15.2:0/64797995 shutdown_connections 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 --2- 172.21.15.2:0/64797995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e941100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 --2- 172.21.15.2:0/64797995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103dc0 0x7f1e94107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 --2- 172.21.15.2:0/64797995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94103410 0x7f1e941037f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 -- 172.21.15.2:0/64797995 >> 172.21.15.2:0/64797995 conn(0x7f1e940fec90 msgr2=0x7f1e941010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 -- 172.21.15.2:0/64797995 shutdown_connections 2024-03-11T18:35:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.194+0000 7f1e93e00640 1 -- 172.21.15.2:0/64797995 wait complete. 2024-03-11T18:35:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.195+0000 7f1e93e00640 1 Processor -- start 2024-03-11T18:35:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.195+0000 7f1e93e00640 1 -- start start 2024-03-11T18:35:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103410 0x7f1e9419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9419a250 con 0x7f1e94103410 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9419a3c0 con 0x7f1e94199780 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9419a580 con 0x7f1e941084e0 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e8be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e8be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:49158/0 (socket says 172.21.15.2:49158) 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42432/0 (socket says 172.21.15.2:42432) 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e8be00640 1 -- 172.21.15.2:0/2290716443 learned_addr learned my addr 172.21.15.2:0/2290716443 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.196+0000 7f1e92a00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103410 0x7f1e9419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 -- 172.21.15.2:0/2290716443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 msgr2=0x7f1e94199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 -- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103410 msgr2=0x7f1e9419f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103410 0x7f1e9419f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 -- 172.21.15.2:0/2290716443 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84009560 con 0x7f1e941084e0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e93400640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8be00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00e9c0 tx=0x7f1e7c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8b400640 1 -- 172.21.15.2:0/2290716443 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c009700 con 0x7f1e941084e0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e941a64d0 con 0x7f1e941084e0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8b400640 1 -- 172.21.15.2:0/2290716443 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e7c00cd80 con 0x7f1e941084e0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e8b400640 1 -- 172.21.15.2:0/2290716443 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c010430 con 0x7f1e941084e0 2024-03-11T18:35:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.197+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e941a6a00 con 0x7f1e941084e0 2024-03-11T18:35:19.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.201+0000 7f1e8b400640 1 -- 172.21.15.2:0/2290716443 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f1e7c005000 con 0x7f1e941084e0 2024-03-11T18:35:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.201+0000 7f1e9c643740 1 Processor -- start 2024-03-11T18:35:19.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 -- start start 2024-03-11T18:35:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d354eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d354f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d355a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d350a620 con 0x5636d354eff0 2024-03-11T18:35:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d34bc8c0 con 0x5636d3558570 2024-03-11T18:35:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e9c643740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d34b9df0 con 0x5636d354fd60 2024-03-11T18:35:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e93400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d354eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d354f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.202+0000 7f1e8be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d355a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e93400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d354eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:49164/0 (socket says 172.21.15.2:49164) 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e93400640 1 -- 172.21.15.2:0/1534520713 learned_addr learned my addr 172.21.15.2:0/1534520713 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e8b400640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 0x7f1e780a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 -- 172.21.15.2:0/1534520713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 msgr2=0x5636d354eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d354eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 -- 172.21.15.2:0/1534520713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 msgr2=0x5636d355a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d355a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 -- 172.21.15.2:0/1534520713 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636d3436b00 con 0x5636d354eff0 2024-03-11T18:35:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e8be00640 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d355a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e8b400640 1 -- 172.21.15.2:0/2290716443 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f1e7c014070 con 0x7f1e941084e0 2024-03-11T18:35:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e93400640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 0x7f1e780a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.203+0000 7f1e92a00640 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d354f470 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1e84036040 tx=0x7f1e8402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20b1588433c28d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.204+0000 7f1e83e00640 1 -- 172.21.15.2:0/1534520713 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8402dea0 con 0x5636d354eff0 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.204+0000 7f1e83e00640 1 -- 172.21.15.2:0/1534520713 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e84004880 con 0x5636d354eff0 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.204+0000 7f1e83e00640 1 -- 172.21.15.2:0/1534520713 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84005050 con 0x5636d354eff0 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- 172.21.15.2:0/1534520713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 msgr2=0x5636d354f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e93400640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 0x7f1e780a40c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009fd0 tx=0x7f1e8c004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d354f470 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1e84036040 tx=0x7f1e8402c640 comp rx=0 tx=0).stop 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- 172.21.15.2:0/1534520713 shutdown_connections 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d355a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d354f470 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 --2- 172.21.15.2:0/1534520713 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d354eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- 172.21.15.2:0/1534520713 >> 172.21.15.2:0/1534520713 conn(0x5636d35164b0 msgr2=0x5636d3549ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- 172.21.15.2:0/1534520713 shutdown_connections 2024-03-11T18:35:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- 172.21.15.2:0/1534520713 wait complete. 2024-03-11T18:35:19.216 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.205+0000 7f1e9c643740 1 -- start start 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 Processor -- start 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d3555510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d3558570 0x5636d3555f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d34b55d0 con 0x5636d354eff0 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d345ab50 con 0x5636d354fd60 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d345a7f0 con 0x5636d3558570 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e9c643740 10 client.0 _ll_register_callbacks cb 0x5636d346e550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e8be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e8be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42458/0 (socket says 172.21.15.2:42458) 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e93400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d3558570 0x5636d3555f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.217 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e8be00640 1 -- 172.21.15.2:0/537099558 learned_addr learned my addr 172.21.15.2:0/537099558 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.206+0000 7f1e92a00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d3555510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 -- 172.21.15.2:0/537099558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d3558570 msgr2=0x5636d3555f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d3558570 0x5636d3555f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 -- 172.21.15.2:0/537099558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 msgr2=0x5636d3555510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d3555510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 -- 172.21.15.2:0/537099558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e941108c0 con 0x5636d354fd60 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e92a00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d3555510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e8be00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c09a740 tx=0x7f1e7c00e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7cb07e5e5beba62 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e80c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c0cb750 con 0x5636d354fd60 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f1e7c09aeb0 con 0x5636d354fd60 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636d3487cd0 con 0x5636d354fd60 2024-03-11T18:35:19.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.207+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e7c09a2f0 con 0x5636d354fd60 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 10 client.25607 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5636d34b3c50 con 0x5636d354fd60 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5636d3487cd0 con 0x5636d354fd60 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f1e7c0d1020 con 0x5636d354fd60 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 10 client.25607 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 10 client.25607 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 20 client.25607 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35684c0 0x5636d356a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 4 client.25607 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.208+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636d34abce0 con 0x5636d35684c0 2024-03-11T18:35:19.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.209+0000 7f1e92a00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35684c0 0x5636d356a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.209+0000 7f1e92a00640 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35684c0 0x5636d356a920 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8849c28c23e237cf server_cookie=8352eedce636337c in_seq=0 out_seq=0 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.209+0000 7f1e80c00640 10 client.25607 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e80c00640 1 -- 172.21.15.2:0/537099558 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5636d34abce0 con 0x5636d35684c0 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e80c00640 10 client.25607 handle_command_reply: tid=0 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 client.25607 shutdown 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 msgr2=0x5636d3555a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c09a740 tx=0x7f1e7c00e940 comp rx=0 tx=0).stop 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 shutdown_connections 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35684c0 0x5636d356a920 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d3558570 0x5636d3555f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d354fd60 0x5636d3555a50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e80c00640 0 client.25607 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 --2- 172.21.15.2:0/537099558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d3555510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 >> 172.21.15.2:0/537099558 conn(0x5636d35164b0 msgr2=0x5636d354ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 shutdown_connections 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.210+0000 7f1e9c643740 1 -- 172.21.15.2:0/537099558 wait complete. 2024-03-11T18:35:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.211+0000 7f1e9c643740 20 client.25607 trim_cache size 0 max 16384 2024-03-11T18:35:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 Processor -- start 2024-03-11T18:35:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 -- start start 2024-03-11T18:35:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d35527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d3552d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d31fde60 con 0x5636d354eff0 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d34abce0 con 0x5636d3558570 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e9c643740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d3487cd0 con 0x5636d354fd60 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42470/0 (socket says 172.21.15.2:42470) 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e93400640 1 -- 172.21.15.2:0/610234719 learned_addr learned my addr 172.21.15.2:0/610234719 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.227+0000 7f1e92a00640 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d35527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e8be00640 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d3552d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 -- 172.21.15.2:0/610234719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 msgr2=0x5636d3552d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d3552d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 -- 172.21.15.2:0/610234719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 msgr2=0x5636d35527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d35527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 -- 172.21.15.2:0/610234719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636d3450530 con 0x5636d3558570 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e93400640 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00fc80 tx=0x7f1e8c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1128e830e4d70a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.228+0000 7f1e83400640 1 -- 172.21.15.2:0/610234719 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c00fcb0 con 0x5636d3558570 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e83400640 1 -- 172.21.15.2:0/610234719 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e8c017040 con 0x5636d3558570 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e83400640 1 -- 172.21.15.2:0/610234719 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c0283f0 con 0x5636d3558570 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 -- 172.21.15.2:0/610234719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 msgr2=0x5636d35484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00fc80 tx=0x7f1e8c01a000 comp rx=0 tx=0).stop 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 -- 172.21.15.2:0/610234719 shutdown_connections 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35484b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d3552d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 --2- 172.21.15.2:0/610234719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d35527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 -- 172.21.15.2:0/610234719 >> 172.21.15.2:0/610234719 conn(0x5636d35164b0 msgr2=0x5636d3553e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 -- 172.21.15.2:0/610234719 shutdown_connections 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.229+0000 7f1e9c643740 1 -- 172.21.15.2:0/610234719 wait complete. 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 -- start start 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 Processor -- start 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d355e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d355e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d355ee10 con 0x5636d354eff0 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d355ef80 con 0x5636d3558570 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.230+0000 7f1e9c643740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636d35bef90 con 0x5636d354fd60 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e9c643740 10 client.0 _ll_register_callbacks cb 0x7f1e7c0d1520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d355e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42472/0 (socket says 172.21.15.2:42472) 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d355e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39570/0 (socket says 172.21.15.2:39570) 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 -- 172.21.15.2:0/3573719161 learned_addr learned my addr 172.21.15.2:0/3573719161 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e8be00640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d355e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 -- 172.21.15.2:0/3573719161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 msgr2=0x5636d355e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d355e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 -- 172.21.15.2:0/3573719161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 msgr2=0x5636d355e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d355e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.231+0000 7f1e93400640 1 -- 172.21.15.2:0/3573719161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c00f9c0 con 0x5636d3558570 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e93400640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00fc50 tx=0x7f1e8c0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1146ce2739f5dfa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e80c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c003b70 con 0x5636d3558570 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f1e8c003cd0 con 0x5636d3558570 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636d34dad30 con 0x5636d3558570 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e8c003e90 con 0x5636d3558570 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e9c643740 10 client.25613 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.232+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5636d34762c0 con 0x5636d3558570 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5636d34dad30 con 0x5636d3558570 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f1e8c027070 con 0x5636d3558570 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 10 client.25613 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 20 client.25613 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35c0740 0x5636d35c2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 4 client.25613 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5636d31b5600 con 0x5636d35c0740 2024-03-11T18:35:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.233+0000 7f1e92a00640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35c0740 0x5636d35c2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.234+0000 7f1e92a00640 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35c0740 0x5636d35c2b60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5d1411a22d0edcc server_cookie=ef25a0e8e23b85cb in_seq=0 out_seq=0 2024-03-11T18:35:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.234+0000 7f1e80c00640 10 client.25613 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:19.377 INFO:tasks.workunit.client.0.smithi002.stdout: 1 38400 11.79 MB/sec warmup 19 sec latency 69.003 ms 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e80c00640 1 -- 172.21.15.2:0/3573719161 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5636d31b5600 con 0x5636d35c0740 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e80c00640 10 client.25613 handle_command_reply: tid=0 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 client.25613 shutdown 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 msgr2=0x5636d35bea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c00fc50 tx=0x7f1e8c0042e0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 shutdown_connections 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5636d35c0740 0x5636d35c2b60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5636d3558570 0x5636d35bea50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5636d354fd60 0x5636d355e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e80c00640 0 client.25613 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 --2- 172.21.15.2:0/3573719161 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5636d354eff0 0x5636d355e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 >> 172.21.15.2:0/3573719161 conn(0x5636d35164b0 msgr2=0x5636d35484b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 shutdown_connections 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.471+0000 7f1e9c643740 1 -- 172.21.15.2:0/3573719161 wait complete. 2024-03-11T18:35:19.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.472+0000 7f1e9c643740 20 client.25613 trim_cache size 0 max 16384 2024-03-11T18:35:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:35:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:35:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:35:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:35:19.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.472+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 msgr2=0x7f1e780a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 0x7f1e780a40c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c009fd0 tx=0x7f1e8c004a40 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 msgr2=0x7f1e9419fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00e9c0 tx=0x7f1e7c00ee80 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 shutdown_connections 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e94199780 0x7f1e94199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1e941084e0 0x7f1e9419fbb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1e780a1c00 0x7f1e780a40c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 --2- 172.21.15.2:0/2290716443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e94103410 0x7f1e9419f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 >> 172.21.15.2:0/2290716443 conn(0x7f1e940fec90 msgr2=0x7f1e94100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 shutdown_connections 2024-03-11T18:35:19.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:19.473+0000 7f1e93e00640 1 -- 172.21.15.2:0/2290716443 wait complete. 2024-03-11T18:35:19.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:19 smithi002 ceph-mon[31328]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 312 op/s 2024-03-11T18:35:19.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:19 smithi023 ceph-mon[36406]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 312 op/s 2024-03-11T18:35:19.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:19 smithi165 ceph-mon[36634]: pgmap v51: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 1.5 MiB/s wr, 312 op/s 2024-03-11T18:35:20.377 INFO:tasks.workunit.client.0.smithi002.stdout: 1 39927 11.60 MB/sec warmup 20 sec latency 14.758 ms 2024-03-11T18:35:21.377 INFO:tasks.workunit.client.0.smithi002.stdout: 1 42374 11.70 MB/sec warmup 21 sec latency 40.802 ms 2024-03-11T18:35:21.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:21 smithi002 ceph-mon[31328]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1016 KiB/s wr, 147 op/s 2024-03-11T18:35:21.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:21 smithi023 ceph-mon[36406]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1016 KiB/s wr, 147 op/s 2024-03-11T18:35:21.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:21 smithi165 ceph-mon[36634]: pgmap v52: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 1016 KiB/s wr, 147 op/s 2024-03-11T18:35:22.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 43706 11.47 MB/sec warmup 22 sec latency 11.733 ms 2024-03-11T18:35:23.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 46308 11.66 MB/sec warmup 23 sec latency 8.846 ms 2024-03-11T18:35:23.707 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:23 smithi023 ceph-mon[36406]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 995 KiB/s wr, 137 op/s 2024-03-11T18:35:23.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:23 smithi002 ceph-mon[31328]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 995 KiB/s wr, 137 op/s 2024-03-11T18:35:23.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:23 smithi165 ceph-mon[36634]: pgmap v53: 129 pgs: 129 active+clean; 19 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 995 KiB/s wr, 137 op/s 2024-03-11T18:35:24.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 47575 11.44 MB/sec warmup 24 sec latency 12.227 ms 2024-03-11T18:35:25.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 49891 11.53 MB/sec warmup 25 sec latency 12.484 ms 2024-03-11T18:35:25.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:25 smithi002 ceph-mon[31328]: pgmap v54: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-11T18:35:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:25 smithi165 ceph-mon[36634]: pgmap v54: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-11T18:35:25.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:25 smithi023 ceph-mon[36406]: pgmap v54: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.5 MiB/s wr, 364 op/s 2024-03-11T18:35:26.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 50922 11.18 MB/sec warmup 26 sec latency 21.653 ms 2024-03-11T18:35:27.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 53079 11.23 MB/sec warmup 27 sec latency 54.053 ms 2024-03-11T18:35:27.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:27 smithi002 ceph-mon[31328]: pgmap v55: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1016 KiB/s wr, 296 op/s 2024-03-11T18:35:27.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:27 smithi023 ceph-mon[36406]: pgmap v55: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1016 KiB/s wr, 296 op/s 2024-03-11T18:35:27.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:27 smithi165 ceph-mon[36634]: pgmap v55: 129 pgs: 129 active+clean; 15 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 1016 KiB/s wr, 296 op/s 2024-03-11T18:35:28.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 54180 11.02 MB/sec warmup 28 sec latency 14.745 ms 2024-03-11T18:35:29.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 56501 11.11 MB/sec warmup 29 sec latency 12.102 ms 2024-03-11T18:35:29.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:29 smithi002 ceph-mon[31328]: pgmap v56: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:35:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:29 smithi165 ceph-mon[36634]: pgmap v56: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:35:29.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:29 smithi023 ceph-mon[36406]: pgmap v56: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:35:30.378 INFO:tasks.workunit.client.0.smithi002.stdout: 1 57590 10.91 MB/sec warmup 30 sec latency 18.110 ms 2024-03-11T18:35:31.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 59707 10.95 MB/sec warmup 31 sec latency 28.843 ms 2024-03-11T18:35:31.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:31 smithi002 ceph-mon[31328]: pgmap v57: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:31.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:31 smithi165 ceph-mon[36634]: pgmap v57: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:31.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:31 smithi023 ceph-mon[36406]: pgmap v57: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:32.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 60462 10.77 MB/sec warmup 32 sec latency 384.098 ms 2024-03-11T18:35:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 -- 172.21.15.2:0/3012757014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe164100a10 msgr2=0x7fe164104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.673 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 --2- 172.21.15.2:0/3012757014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe164100a10 0x7fe164104ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe154009900 tx=0x7fe15402bd30 comp rx=0 tx=0).stop 2024-03-11T18:35:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 -- 172.21.15.2:0/3012757014 shutdown_connections 2024-03-11T18:35:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 --2- 172.21.15.2:0/3012757014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1641054f0 0x7fe164105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 --2- 172.21.15.2:0/3012757014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe164100a10 0x7fe164104ed0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 --2- 172.21.15.2:0/3012757014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe1641004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.674 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.672+0000 7fe169600640 1 -- 172.21.15.2:0/3012757014 >> 172.21.15.2:0/3012757014 conn(0x7fe1640fbb20 msgr2=0x7fe1640fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.673+0000 7fe169600640 1 -- 172.21.15.2:0/3012757014 shutdown_connections 2024-03-11T18:35:32.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.673+0000 7fe169600640 1 -- 172.21.15.2:0/3012757014 wait complete. 2024-03-11T18:35:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.674+0000 7fe169600640 1 Processor -- start 2024-03-11T18:35:32.676 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 -- start start 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 0x7fe16419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1641054f0 0x7fe1641a26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641a2cc0 con 0x7fe164100a10 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641a2e30 con 0x7fe1641054f0 2024-03-11T18:35:32.677 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.675+0000 7fe169600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641a2ff0 con 0x7fe1641000f0 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:37570/0 (socket says 172.21.15.2:37570) 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 -- 172.21.15.2:0/3608506378 learned_addr learned my addr 172.21.15.2:0/3608506378 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe163400640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1641054f0 0x7fe1641a26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162000640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 0x7fe16419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 -- 172.21.15.2:0/3608506378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1641054f0 msgr2=0x7fe1641a26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1641054f0 0x7fe1641a26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 -- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 msgr2=0x7fe16419e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 0x7fe16419e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162a00640 1 -- 172.21.15.2:0/3608506378 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154009560 con 0x7fe1641000f0 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.676+0000 7fe162000640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 0x7fe16419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:35:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe162a00640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b450 tx=0x7fe15000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe15b400640 1 -- 172.21.15.2:0/3608506378 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150011890 con 0x7fe1641000f0 2024-03-11T18:35:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1641a32d0 con 0x7fe1641000f0 2024-03-11T18:35:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe15b400640 1 -- 172.21.15.2:0/3608506378 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1500103e0 con 0x7fe1641000f0 2024-03-11T18:35:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe15b400640 1 -- 172.21.15.2:0/3608506378 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15000f5a0 con 0x7fe1641000f0 2024-03-11T18:35:32.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.677+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1641aab50 con 0x7fe1641000f0 2024-03-11T18:35:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 Processor -- start 2024-03-11T18:35:32.683 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 -- start start 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d1675230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d1680720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1630620 con 0x5563d167e330 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d15e28c0 con 0x5563d1674db0 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.682+0000 7fe16c384740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d15dfdf0 con 0x5563d1675b20 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.683+0000 7fe163400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.683+0000 7fe163400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:37572/0 (socket says 172.21.15.2:37572) 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.683+0000 7fe163400640 1 -- 172.21.15.2:0/2146238393 learned_addr learned my addr 172.21.15.2:0/2146238393 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.683+0000 7fe15b400640 1 -- 172.21.15.2:0/3608506378 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fe150010550 con 0x7fe1641000f0 2024-03-11T18:35:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.684+0000 7fe162a00640 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d1675230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.684+0000 7fe162a00640 1 -- 172.21.15.2:0/2146238393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 msgr2=0x5563d1674870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe162a00640 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe162a00640 1 -- 172.21.15.2:0/2146238393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 msgr2=0x5563d1680720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe162a00640 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d1680720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe162a00640 1 -- 172.21.15.2:0/2146238393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563d155cb00 con 0x5563d1674db0 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe15b400640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 0x7fe1480a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe163400640 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.685+0000 7fe15b400640 1 -- 172.21.15.2:0/3608506378 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe1500cd9a0 con 0x7fe1641000f0 2024-03-11T18:35:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.686+0000 7fe163400640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 0x7fe1480a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.687+0000 7fe162a00640 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d1675230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe1500cae00 tx=0x7fe1500d3e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619b457dab72fb24 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.687+0000 7fe14f400640 1 -- 172.21.15.2:0/2146238393 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15009a0b0 con 0x5563d1674db0 2024-03-11T18:35:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.687+0000 7fe14f400640 1 -- 172.21.15.2:0/2146238393 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1500d3020 con 0x5563d1674db0 2024-03-11T18:35:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.687+0000 7fe14f400640 1 -- 172.21.15.2:0/2146238393 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15009a2e0 con 0x5563d1674db0 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 -- 172.21.15.2:0/2146238393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 msgr2=0x5563d1675230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d1675230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe1500cae00 tx=0x7fe1500d3e30 comp rx=0 tx=0).stop 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe163400640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 0x7fe1480a40a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011630 tx=0x7fe15c007740 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 -- 172.21.15.2:0/2146238393 shutdown_connections 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d1680720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d1675230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 --2- 172.21.15.2:0/2146238393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d1674870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 -- 172.21.15.2:0/2146238393 >> 172.21.15.2:0/2146238393 conn(0x5563d163c4b0 msgr2=0x5563d166faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.688+0000 7fe16c384740 1 -- 172.21.15.2:0/2146238393 shutdown_connections 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.689+0000 7fe16c384740 1 -- 172.21.15.2:0/2146238393 wait complete. 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.689+0000 7fe16c384740 1 -- start start 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.689+0000 7fe16c384740 1 Processor -- start 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d167b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d167b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d167bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d15db5d0 con 0x5563d167e330 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1580b50 con 0x5563d1674db0 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d15807f0 con 0x5563d1675b20 2024-03-11T18:35:32.692 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.690+0000 7fe16c384740 10 client.0 _ll_register_callbacks cb 0x5563d1594550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.691+0000 7fe162a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d167b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.691+0000 7fe163400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d167bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.691+0000 7fe163400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d167bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54790/0 (socket says 172.21.15.2:54790) 2024-03-11T18:35:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.691+0000 7fe163400640 1 -- 172.21.15.2:0/4163766142 learned_addr learned my addr 172.21.15.2:0/4163766142 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.691+0000 7fe162000640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d167b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 -- 172.21.15.2:0/4163766142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 msgr2=0x5563d167b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d167b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 -- 172.21.15.2:0/4163766142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 msgr2=0x5563d167bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d167bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 -- 172.21.15.2:0/4163766142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1641aab50 con 0x5563d1674db0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.692+0000 7fe162a00640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d167b340 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1500cae60 tx=0x7fe150010d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb436f15a3592ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.693+0000 7fe143e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.693+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1500d3020 con 0x5563d1674db0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.693+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe15009e430 con 0x5563d1674db0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.693+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563d15adcd0 con 0x5563d1674db0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.693+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe15009aa50 con 0x5563d1674db0 2024-03-11T18:35:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.694+0000 7fe16c384740 10 client.25625 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.694+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5563d15d9c50 con 0x5563d1674db0 2024-03-11T18:35:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.695+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe150063020 con 0x5563d1674db0 2024-03-11T18:35:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.695+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe1500caf70 con 0x5563d1674db0 2024-03-11T18:35:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.695+0000 7fe16c384740 10 client.25625 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.695+0000 7fe16c384740 10 client.25625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.695+0000 7fe16c384740 20 client.25625 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.696+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d168e020 0x5563d1690410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.696+0000 7fe16c384740 4 client.25625 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:35:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.696+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563d15d1ce0 con 0x5563d168e020 2024-03-11T18:35:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.696+0000 7fe162000640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d168e020 0x5563d1690410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.697+0000 7fe162000640 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d168e020 0x5563d1690410 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2042458e3cc0696f server_cookie=edecba1969ac9e0f in_seq=0 out_seq=0 2024-03-11T18:35:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.697+0000 7fe143e00640 10 client.25625 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe143e00640 1 -- 172.21.15.2:0/4163766142 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5563d15d1ce0 con 0x5563d168e020 2024-03-11T18:35:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe143e00640 10 client.25625 handle_command_reply: tid=0 2024-03-11T18:35:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe16c384740 1 client.25625 shutdown 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 msgr2=0x5563d167b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d167b340 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1500cae60 tx=0x7fe150010d00 comp rx=0 tx=0).stop 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.698+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 shutdown_connections 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d168e020 0x5563d1690410 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d167e330 0x5563d167bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d167b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 --2- 172.21.15.2:0/4163766142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d1674db0 0x5563d167b340 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 >> 172.21.15.2:0/4163766142 conn(0x5563d163c4b0 msgr2=0x5563d16717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe143e00640 0 client.25625 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 shutdown_connections 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 1 -- 172.21.15.2:0/4163766142 wait complete. 2024-03-11T18:35:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.699+0000 7fe16c384740 20 client.25625 trim_cache size 0 max 16384 2024-03-11T18:35:32.729 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.727+0000 7fe16c384740 1 Processor -- start 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 -- start start 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 0x5563d167bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16851b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1323e60 con 0x5563d1675b20 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1680f40 con 0x5563d167e330 2024-03-11T18:35:32.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.728+0000 7fe16c384740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1674080 con 0x5563d1674db0 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:37600/0 (socket says 172.21.15.2:37600) 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 -- 172.21.15.2:0/1630949037 learned_addr learned my addr 172.21.15.2:0/1630949037 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe163400640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16851b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162000640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 0x5563d167bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 -- 172.21.15.2:0/1630949037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 msgr2=0x5563d16851b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16851b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 -- 172.21.15.2:0/1630949037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 msgr2=0x5563d167bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 0x5563d167bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.729+0000 7fe162a00640 1 -- 172.21.15.2:0/1630949037 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563d1576530 con 0x5563d1674db0 2024-03-11T18:35:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.730+0000 7fe162000640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 0x5563d167bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:35:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.731+0000 7fe162a00640 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe150098590 tx=0x7fe1500d3e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e7b7d61e100a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.731+0000 7fe14ea00640 1 -- 172.21.15.2:0/1630949037 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe150099e20 con 0x5563d1674db0 2024-03-11T18:35:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.731+0000 7fe14ea00640 1 -- 172.21.15.2:0/1630949037 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe150063070 con 0x5563d1674db0 2024-03-11T18:35:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.732+0000 7fe14ea00640 1 -- 172.21.15.2:0/1630949037 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15009a2e0 con 0x5563d1674db0 2024-03-11T18:35:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.733+0000 7fe16c384740 1 -- 172.21.15.2:0/1630949037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 msgr2=0x5563d167da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.735 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe150098590 tx=0x7fe1500d3e30 comp rx=0 tx=0).stop 2024-03-11T18:35:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 -- 172.21.15.2:0/1630949037 shutdown_connections 2024-03-11T18:35:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16851b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1675b20 0x5563d167bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 --2- 172.21.15.2:0/1630949037 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1674db0 0x5563d167da70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.734+0000 7fe16c384740 1 -- 172.21.15.2:0/1630949037 >> 172.21.15.2:0/1630949037 conn(0x5563d163c4b0 msgr2=0x5563d166f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.735+0000 7fe16c384740 1 -- 172.21.15.2:0/1630949037 shutdown_connections 2024-03-11T18:35:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.735+0000 7fe16c384740 1 -- 172.21.15.2:0/1630949037 wait complete. 2024-03-11T18:35:32.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.736+0000 7fe16c384740 1 -- start start 2024-03-11T18:35:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.736+0000 7fe16c384740 1 Processor -- start 2024-03-11T18:35:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1674db0 0x5563d1685450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d16df570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d16e0070 con 0x5563d1674db0 2024-03-11T18:35:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d16e01e0 con 0x5563d167e330 2024-03-11T18:35:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563d1677c50 con 0x5563d1675b20 2024-03-11T18:35:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe16c384740 10 client.0 _ll_register_callbacks cb 0x7fe15c0075e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe163400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe162000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d16df570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe163400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42786/0 (socket says 172.21.15.2:42786) 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.737+0000 7fe163400640 1 -- 172.21.15.2:0/2308230368 learned_addr learned my addr 172.21.15.2:0/2308230368 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe162a00640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1674db0 0x5563d1685450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 -- 172.21.15.2:0/2308230368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 msgr2=0x5563d16df570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d16df570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 -- 172.21.15.2:0/2308230368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1674db0 msgr2=0x5563d1685450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1674db0 0x5563d1685450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 -- 172.21.15.2:0/2308230368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563d15d9c50 con 0x5563d167e330 2024-03-11T18:35:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.738+0000 7fe163400640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe15c007a30 tx=0x7fe15c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d1bd45c07e5301 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.739+0000 7fe143e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.739+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15c01ee00 con 0x5563d167e330 2024-03-11T18:35:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.739+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563d1600d30 con 0x5563d167e330 2024-03-11T18:35:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.739+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe15c01b400 con 0x5563d167e330 2024-03-11T18:35:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.740+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe15c01b5c0 con 0x5563d167e330 2024-03-11T18:35:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.740+0000 7fe16c384740 10 client.25631 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.740+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5563d159c2c0 con 0x5563d167e330 2024-03-11T18:35:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.740+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5563d1600d30 con 0x5563d167e330 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fe15c0287f0 con 0x5563d167e330 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 10 client.25631 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 10 client.25631 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 20 client.25631 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d16848a0 0x5563d16f94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 4 client.25631 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5563d12db600 con 0x5563d16848a0 2024-03-11T18:35:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.741+0000 7fe162a00640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d16848a0 0x5563d16f94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.743+0000 7fe162a00640 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d16848a0 0x5563d16f94e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14eeef9d51af3327 server_cookie=1aa68d9a6e9b5b26 in_seq=0 out_seq=0 2024-03-11T18:35:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.743+0000 7fe143e00640 10 client.25631 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe143e00640 1 -- 172.21.15.2:0/2308230368 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5563d12db600 con 0x5563d16848a0 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe143e00640 10 client.25631 handle_command_reply: tid=0 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 client.25631 shutdown 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 msgr2=0x5563d16dfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe15c007a30 tx=0x7fe15c004a40 comp rx=0 tx=0).stop 2024-03-11T18:35:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 shutdown_connections 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5563d16848a0 0x5563d16f94e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563d167e330 0x5563d16dfab0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5563d1675b20 0x5563d16df570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 --2- 172.21.15.2:0/2308230368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5563d1674db0 0x5563d1685450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 >> 172.21.15.2:0/2308230368 conn(0x5563d163c4b0 msgr2=0x5563d1679ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.881+0000 7fe143e00640 0 client.25631 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.882+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 shutdown_connections 2024-03-11T18:35:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.882+0000 7fe16c384740 1 -- 172.21.15.2:0/2308230368 wait complete. 2024-03-11T18:35:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.882+0000 7fe16c384740 20 client.25631 trim_cache size 0 max 16384 2024-03-11T18:35:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:35:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:35:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:35:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.883+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 msgr2=0x7fe1480a40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.883+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 0x7fe1480a40a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe15c011630 tx=0x7fe15c007740 comp rx=0 tx=0).stop 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.883+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 msgr2=0x7fe16419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.883+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe15000b450 tx=0x7fe15000b910 comp rx=0 tx=0).stop 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 shutdown_connections 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1641054f0 0x7fe1641a26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1480a1be0 0x7fe1480a40a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe164100a10 0x7fe16419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 --2- 172.21.15.2:0/3608506378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe1641000f0 0x7fe16419de00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 >> 172.21.15.2:0/3608506378 conn(0x7fe1640fbb20 msgr2=0x7fe1640fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 shutdown_connections 2024-03-11T18:35:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:32.884+0000 7fe169600640 1 -- 172.21.15.2:0/3608506378 wait complete. 2024-03-11T18:35:33.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 61129 10.50 MB/sec warmup 33 sec latency 51.624 ms 2024-03-11T18:35:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:33 smithi002 ceph-mon[31328]: pgmap v58: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:33.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:33 smithi023 ceph-mon[36406]: pgmap v58: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:33 smithi165 ceph-mon[36634]: pgmap v58: 129 pgs: 129 active+clean; 39 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 2.6 MiB/s wr, 292 op/s 2024-03-11T18:35:34.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 63308 10.55 MB/sec warmup 34 sec latency 12.574 ms 2024-03-11T18:35:35.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 64614 10.44 MB/sec warmup 35 sec latency 15.859 ms 2024-03-11T18:35:35.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:35 smithi002 ceph-mon[31328]: pgmap v59: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 6.3 MiB/s wr, 550 op/s 2024-03-11T18:35:35.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:35 smithi023 ceph-mon[36406]: pgmap v59: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 6.3 MiB/s wr, 550 op/s 2024-03-11T18:35:35.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:35 smithi165 ceph-mon[36634]: pgmap v59: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 6.3 MiB/s wr, 550 op/s 2024-03-11T18:35:36.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 66639 10.45 MB/sec warmup 36 sec latency 29.725 ms 2024-03-11T18:35:37.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 68143 10.39 MB/sec warmup 37 sec latency 13.777 ms 2024-03-11T18:35:37.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:37 smithi002 ceph-mon[31328]: pgmap v60: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 5.8 MiB/s wr, 322 op/s 2024-03-11T18:35:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:37 smithi165 ceph-mon[36634]: pgmap v60: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 5.8 MiB/s wr, 322 op/s 2024-03-11T18:35:37.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:37 smithi023 ceph-mon[36406]: pgmap v60: 129 pgs: 129 active+clean; 32 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 5.8 MiB/s wr, 322 op/s 2024-03-11T18:35:38.379 INFO:tasks.workunit.client.0.smithi002.stdout: 1 69927 10.32 MB/sec warmup 38 sec latency 18.128 ms 2024-03-11T18:35:39.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 71423 10.34 MB/sec warmup 39 sec latency 10.909 ms 2024-03-11T18:35:39.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:39 smithi002 ceph-mon[31328]: pgmap v61: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.2 MiB/s wr, 393 op/s 2024-03-11T18:35:39.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:39 smithi165 ceph-mon[36634]: pgmap v61: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.2 MiB/s wr, 393 op/s 2024-03-11T18:35:39.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:39 smithi023 ceph-mon[36406]: pgmap v61: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 6.2 MiB/s wr, 393 op/s 2024-03-11T18:35:40.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 72858 10.22 MB/sec warmup 40 sec latency 15.034 ms 2024-03-11T18:35:41.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 74796 10.30 MB/sec warmup 41 sec latency 13.430 ms 2024-03-11T18:35:41.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:41 smithi023 ceph-mon[36406]: pgmap v62: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:41.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:41 smithi165 ceph-mon[36634]: pgmap v62: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:41 smithi002 ceph-mon[31328]: pgmap v62: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:42.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 75727 10.11 MB/sec warmup 42 sec latency 17.249 ms 2024-03-11T18:35:43.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 78070 10.23 MB/sec warmup 43 sec latency 30.219 ms 2024-03-11T18:35:43.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:43 smithi165 ceph-mon[36634]: pgmap v63: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:43.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:43 smithi023 ceph-mon[36406]: pgmap v63: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:44.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:43 smithi002 ceph-mon[31328]: pgmap v63: 129 pgs: 129 active+clean; 32 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 4.1 MiB/s wr, 328 op/s 2024-03-11T18:35:44.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 78989 10.05 MB/sec warmup 44 sec latency 18.376 ms 2024-03-11T18:35:45.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 81005 10.09 MB/sec warmup 45 sec latency 12.632 ms 2024-03-11T18:35:45.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:45 smithi165 ceph-mon[36634]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 384 op/s 2024-03-11T18:35:45.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:45 smithi023 ceph-mon[36406]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 384 op/s 2024-03-11T18:35:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:45 smithi002 ceph-mon[31328]: pgmap v64: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 384 op/s 2024-03-11T18:35:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.048+0000 7f71ab200640 1 -- 172.21.15.2:0/2900006645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac103dc0 msgr2=0x7f71ac107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.048+0000 7f71ab200640 1 --2- 172.21.15.2:0/2900006645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac103dc0 0x7f71ac107e00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7194009c10 tx=0x7f719402bee0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 -- 172.21.15.2:0/2900006645 shutdown_connections 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 --2- 172.21.15.2:0/2900006645 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac1084e0 0x7f71ac1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 --2- 172.21.15.2:0/2900006645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac103dc0 0x7f71ac107e00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 --2- 172.21.15.2:0/2900006645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac103410 0x7f71ac1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 -- 172.21.15.2:0/2900006645 >> 172.21.15.2:0/2900006645 conn(0x7f71ac0fec90 msgr2=0x7f71ac1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 -- 172.21.15.2:0/2900006645 shutdown_connections 2024-03-11T18:35:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.049+0000 7f71ab200640 1 -- 172.21.15.2:0/2900006645 wait complete. 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.050+0000 7f71ab200640 1 Processor -- start 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.050+0000 7f71ab200640 1 -- start start 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.050+0000 7f71ab200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71ab200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71ab200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac1996f0 0x7f71ac199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71ab200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19a1c0 con 0x7f71ac1996f0 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71ab200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19a330 con 0x7f71ac1084e0 2024-03-11T18:35:46.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71ab200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac19a4f0 con 0x7f71ac103410 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58114/0 (socket says 172.21.15.2:58114) 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58176/0 (socket says 172.21.15.2:58176) 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9e00640 1 -- 172.21.15.2:0/2209508371 learned_addr learned my addr 172.21.15.2:0/2209508371 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71aa800640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac1996f0 0x7f71ac199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 -- 172.21.15.2:0/2209508371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 msgr2=0x7f71ac19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 -- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac1996f0 msgr2=0x7f71ac199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac1996f0 0x7f71ac199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9400640 1 -- 172.21.15.2:0/2209508371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7194009840 con 0x7f71ac1084e0 2024-03-11T18:35:46.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.051+0000 7f71a9e00640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71a9400640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f719402c4f0 tx=0x7f7194033730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71a2a00640 1 -- 172.21.15.2:0/2209508371 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194003b80 con 0x7f71ac1084e0 2024-03-11T18:35:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71a2a00640 1 -- 172.21.15.2:0/2209508371 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7194003ce0 con 0x7f71ac1084e0 2024-03-11T18:35:46.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71a2a00640 1 -- 172.21.15.2:0/2209508371 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719403e920 con 0x7f71ac1084e0 2024-03-11T18:35:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac1a63e0 con 0x7f71ac1084e0 2024-03-11T18:35:46.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.052+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac1a6870 con 0x7f71ac1084e0 2024-03-11T18:35:46.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.055+0000 7f71a2a00640 1 -- 172.21.15.2:0/2209508371 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f719403ea80 con 0x7f71ac1084e0 2024-03-11T18:35:46.059 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.056+0000 7f71b3bd5740 1 Processor -- start 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.056+0000 7f71b3bd5740 1 -- start start 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363facc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fa7c620 con 0x56363faca570 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fa2e8c0 con 0x56363fac0ff0 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fa2bdf0 con 0x56363fac1d60 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.060 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58180/0 (socket says 172.21.15.2:58180) 2024-03-11T18:35:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71a9e00640 1 -- 172.21.15.2:0/404812267 learned_addr learned my addr 172.21.15.2:0/404812267 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71aa800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.057+0000 7f71a2a00640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 0x7f718c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a2a00640 1 -- 172.21.15.2:0/2209508371 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f71940f0850 con 0x7f71ac1084e0 2024-03-11T18:35:46.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9400640 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363facc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71aa800640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 0x7f718c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 -- 172.21.15.2:0/404812267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 msgr2=0x56363fac0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 -- 172.21.15.2:0/404812267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 msgr2=0x56363facc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363facc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 -- 172.21.15.2:0/404812267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56363f9a8b00 con 0x56363fac0ff0 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9400640 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363facc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:35:46.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f71a9e00640 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f719c0115c0 tx=0x7f719c011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d6c2ad8db9489 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f719b400640 1 -- 172.21.15.2:0/404812267 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c00f6c0 con 0x56363fac0ff0 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f719b400640 1 -- 172.21.15.2:0/404812267 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f719c00fd00 con 0x56363fac0ff0 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.058+0000 7f719b400640 1 -- 172.21.15.2:0/404812267 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c019e90 con 0x56363fac0ff0 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 -- 172.21.15.2:0/404812267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 msgr2=0x56363fac1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f719c0115c0 tx=0x7f719c011a80 comp rx=0 tx=0).stop 2024-03-11T18:35:46.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71aa800640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 0x7f718c0a40c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b610 tx=0x7f71a4004730 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 -- 172.21.15.2:0/404812267 shutdown_connections 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363facc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac1470 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/404812267 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 -- 172.21.15.2:0/404812267 >> 172.21.15.2:0/404812267 conn(0x56363fa884b0 msgr2=0x56363fabbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 -- 172.21.15.2:0/404812267 shutdown_connections 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.059+0000 7f71b3bd5740 1 -- 172.21.15.2:0/404812267 wait complete. 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.060+0000 7f71b3bd5740 1 -- start start 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.060+0000 7f71b3bd5740 1 Processor -- start 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac0ff0 0x56363fac7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363fac8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fa275d0 con 0x56363faca570 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363f9ccb50 con 0x56363fac1d60 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363f9cc7f0 con 0x56363fac0ff0 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71b3bd5740 10 client.0 _ll_register_callbacks cb 0x56363f9e0550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58184/0 (socket says 172.21.15.2:58184) 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 -- 172.21.15.2:0/4083235901 learned_addr learned my addr 172.21.15.2:0/4083235901 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:46.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71aa800640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363fac8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9e00640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac0ff0 0x56363fac7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 -- 172.21.15.2:0/4083235901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac0ff0 msgr2=0x56363fac7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac0ff0 0x56363fac7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 -- 172.21.15.2:0/4083235901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 msgr2=0x56363fac8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363fac8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.061+0000 7f71a9400640 1 -- 172.21.15.2:0/4083235901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c009db0 con 0x56363fac1d60 2024-03-11T18:35:46.072 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f71a9400640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7194000f80 tx=0x7f71940bcdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e10f13c9c6a25c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f7193e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940c1120 con 0x56363fac1d60 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f71940c12e0 con 0x56363fac1d60 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56363f9f9cd0 con 0x56363fac1d60 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71940ee020 con 0x56363fac1d60 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f71b3bd5740 10 client.25643 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.062+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56363fa25c50 con 0x56363fac1d60 2024-03-11T18:35:46.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56363f9f9cd0 con 0x56363fac1d60 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f719403c070 con 0x56363fac1d60 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 10 client.25643 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 10 client.25643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 20 client.25643 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fada440 0x56363fadc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.074 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 4 client.25643 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:35:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56363fa1dce0 con 0x56363fada440 2024-03-11T18:35:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.063+0000 7f71a9e00640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fada440 0x56363fadc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.064+0000 7f71a9e00640 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fada440 0x56363fadc860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cf96fe7e469a89 server_cookie=ac4254adbc840089 in_seq=0 out_seq=0 2024-03-11T18:35:46.075 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.064+0000 7f7193e00640 10 client.25643 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f7193e00640 1 -- 172.21.15.2:0/4083235901 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56363fa1dce0 con 0x56363fada440 2024-03-11T18:35:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f7193e00640 10 client.25643 handle_command_reply: tid=0 2024-03-11T18:35:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 client.25643 shutdown 2024-03-11T18:35:46.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 msgr2=0x56363fac7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7194000f80 tx=0x7f71940bcdc0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 shutdown_connections 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fada440 0x56363fadc860 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363faca570 0x56363fac8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac1d60 0x56363fac7b40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/4083235901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac0ff0 0x56363fac7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 >> 172.21.15.2:0/4083235901 conn(0x56363fa884b0 msgr2=0x56363fabda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f7193e00640 0 client.25643 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:46.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 shutdown_connections 2024-03-11T18:35:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.066+0000 7f71b3bd5740 1 -- 172.21.15.2:0/4083235901 wait complete. 2024-03-11T18:35:46.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.067+0000 7f71b3bd5740 20 client.25643 trim_cache size 0 max 16384 2024-03-11T18:35:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.083+0000 7f71b3bd5740 1 Processor -- start 2024-03-11T18:35:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.083+0000 7f71b3bd5740 1 -- start start 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 0x56363fac7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 0x56363fad1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363f76fe60 con 0x56363fac1d60 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363f9f9cd0 con 0x56363fac0ff0 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fab73d0 con 0x56363faca570 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58194/0 (socket says 172.21.15.2:58194) 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 -- 172.21.15.2:0/610303141 learned_addr learned my addr 172.21.15.2:0/610303141 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:46.088 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9400640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 0x56363fac7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 -- 172.21.15.2:0/610303141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 msgr2=0x56363fad1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71aa800640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 0x56363fad1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 0x56363fad1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 -- 172.21.15.2:0/610303141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 msgr2=0x56363fac7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.084+0000 7f71a9e00640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 0x56363fac7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f71a9e00640 1 -- 172.21.15.2:0/610303141 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56363f9c2530 con 0x56363fac0ff0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f71a9400640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 0x56363fac7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f71aa800640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 0x56363fad1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f71a9e00640 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f719c0197a0 tx=0x7f719c005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdae45f1022e6103 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f719aa00640 1 -- 172.21.15.2:0/610303141 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c019410 con 0x56363fac0ff0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f719aa00640 1 -- 172.21.15.2:0/610303141 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f719c005590 con 0x56363fac0ff0 2024-03-11T18:35:46.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.085+0000 7f719aa00640 1 -- 172.21.15.2:0/610303141 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c009c50 con 0x56363fac0ff0 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 -- 172.21.15.2:0/610303141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 msgr2=0x56363fac9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f719c0197a0 tx=0x7f719c005040 comp rx=0 tx=0).stop 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 -- 172.21.15.2:0/610303141 shutdown_connections 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363faca570 0x56363fad1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac1d60 0x56363fac7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/610303141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363fac0ff0 0x56363fac9d30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 -- 172.21.15.2:0/610303141 >> 172.21.15.2:0/610303141 conn(0x56363fa884b0 msgr2=0x56363fac5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 -- 172.21.15.2:0/610303141 shutdown_connections 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.086+0000 7f71b3bd5740 1 -- 172.21.15.2:0/610303141 wait complete. 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.087+0000 7f71b3bd5740 1 -- start start 2024-03-11T18:35:46.095 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.087+0000 7f71b3bd5740 1 Processor -- start 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.087+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.087+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.087+0000 7f71b3bd5740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 0x56363fac4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fb30a20 con 0x56363fac0ff0 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fb30b90 con 0x56363faca570 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71b3bd5740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56363fb30d50 con 0x56363fac1d60 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71b3bd5740 10 client.0 _ll_register_callbacks cb 0x7f7194003ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71aa800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 0x56363fac4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33828/0 (socket says 172.21.15.2:33828) 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 -- 172.21.15.2:0/2009814003 learned_addr learned my addr 172.21.15.2:0/2009814003 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9400640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 -- 172.21.15.2:0/2009814003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 msgr2=0x56363fac4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 -- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 msgr2=0x56363fac4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 0x56363fac4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71a9e00640 1 -- 172.21.15.2:0/2009814003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56363f9a8b00 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.088+0000 7f71aa800640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 0x56363fac4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f71a9e00640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f719c00f930 tx=0x7f719c019b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290d46a5a8fdc9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f7193e00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c019410 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f719c019e20 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56363fa4cd30 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f719c0050b0 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f71b3bd5740 10 client.15534 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.089+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56363f9e82c0 con 0x56363fac0ff0 2024-03-11T18:35:46.097 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56363fa4cd30 con 0x56363fac0ff0 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f719c003a90 con 0x56363fac0ff0 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 10 client.15534 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 10 client.15534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 20 client.15534 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fb32100 0x56363fb34520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 4 client.15534 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56363f727600 con 0x56363fb32100 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.090+0000 7f71a9400640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fb32100 0x56363fb34520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.091+0000 7f71a9400640 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fb32100 0x56363fb34520 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7209f5a78f977623 server_cookie=f22c7b664404e9ad in_seq=0 out_seq=0 2024-03-11T18:35:46.098 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.091+0000 7f7193e00640 10 client.15534 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:46.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f7193e00640 1 -- 172.21.15.2:0/2009814003 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56363f727600 con 0x56363fb32100 2024-03-11T18:35:46.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f7193e00640 10 client.15534 handle_command_reply: tid=0 2024-03-11T18:35:46.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 client.15534 shutdown 2024-03-11T18:35:46.297 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 msgr2=0x56363fac4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f719c00f930 tx=0x7f719c019b20 comp rx=0 tx=0).stop 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 shutdown_connections 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56363fb32100 0x56363fb34520 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56363faca570 0x56363fac4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56363fac1d60 0x56363fac4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 --2- 172.21.15.2:0/2009814003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56363fac0ff0 0x56363fac4540 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 >> 172.21.15.2:0/2009814003 conn(0x56363fa884b0 msgr2=0x56363fabcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f7193e00640 0 client.15534 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 shutdown_connections 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.294+0000 7f71b3bd5740 1 -- 172.21.15.2:0/2009814003 wait complete. 2024-03-11T18:35:46.298 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71b3bd5740 20 client.15534 trim_cache size 0 max 16384 2024-03-11T18:35:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:35:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:35:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:35:46.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 msgr2=0x7f718c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 0x7f718c0a40c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71a400b610 tx=0x7f71a4004730 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 msgr2=0x7f71ac19fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f719402c4f0 tx=0x7f7194033730 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 shutdown_connections 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f718c0a1c00 0x7f718c0a40c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f71ac1996f0 0x7f71ac199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71ac1084e0 0x7f71ac19fb20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.295+0000 7f71ab200640 1 --2- 172.21.15.2:0/2209508371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71ac103410 0x7f71ac19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.296+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 >> 172.21.15.2:0/2209508371 conn(0x7f71ac0fec90 msgr2=0x7f71ac100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.296+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 shutdown_connections 2024-03-11T18:35:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:46.296+0000 7f71ab200640 1 -- 172.21.15.2:0/2209508371 wait complete. 2024-03-11T18:35:46.380 INFO:tasks.workunit.client.0.smithi002.stdout: 1 82108 10.02 MB/sec warmup 46 sec latency 146.238 ms 2024-03-11T18:35:47.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 83525 9.92 MB/sec warmup 47 sec latency 16.808 ms 2024-03-11T18:35:47.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:47 smithi165 ceph-mon[36634]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 808 KiB/s wr, 125 op/s 2024-03-11T18:35:47.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:47 smithi023 ceph-mon[36406]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 808 KiB/s wr, 125 op/s 2024-03-11T18:35:48.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:47 smithi002 ceph-mon[31328]: pgmap v65: 129 pgs: 129 active+clean; 35 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 808 KiB/s wr, 125 op/s 2024-03-11T18:35:48.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 85541 10.00 MB/sec warmup 48 sec latency 20.369 ms 2024-03-11T18:35:49.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 87353 9.91 MB/sec warmup 49 sec latency 13.367 ms 2024-03-11T18:35:49.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:49 smithi165 ceph-mon[36634]: pgmap v66: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-11T18:35:49.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:49 smithi023 ceph-mon[36406]: pgmap v66: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-11T18:35:50.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:49 smithi002 ceph-mon[31328]: pgmap v66: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-11T18:35:50.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 89277 9.98 MB/sec warmup 50 sec latency 19.532 ms 2024-03-11T18:35:51.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 91285 9.94 MB/sec warmup 51 sec latency 11.176 ms 2024-03-11T18:35:51.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:51 smithi165 ceph-mon[36634]: pgmap v67: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:51.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:51 smithi023 ceph-mon[36406]: pgmap v67: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:52.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:51 smithi002 ceph-mon[31328]: pgmap v67: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:52.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 92991 9.97 MB/sec warmup 52 sec latency 12.625 ms 2024-03-11T18:35:53.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 95242 10.01 MB/sec warmup 53 sec latency 29.550 ms 2024-03-11T18:35:53.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:53 smithi165 ceph-mon[36634]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:53.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:53 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:35:53.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:53 smithi023 ceph-mon[36406]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:53.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:53 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:35:54.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:53 smithi002 ceph-mon[31328]: pgmap v68: 129 pgs: 129 active+clean; 34 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 771 KiB/s wr, 240 op/s 2024-03-11T18:35:54.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:53 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:35:54.381 INFO:tasks.workunit.client.0.smithi002.stdout: 1 96607 9.96 MB/sec warmup 54 sec latency 11.888 ms 2024-03-11T18:35:55.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 99166 10.03 MB/sec warmup 55 sec latency 10.563 ms 2024-03-11T18:35:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:55 smithi165 ceph-mon[36634]: pgmap v69: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 307 op/s 2024-03-11T18:35:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:55 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:35:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:55 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:35:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:55 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:35:55.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:55 smithi023 ceph-mon[36406]: pgmap v69: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 307 op/s 2024-03-11T18:35:55.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:55 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:35:55.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:55 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:35:55.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:55 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:35:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:55 smithi002 ceph-mon[31328]: pgmap v69: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.2 MiB/s wr, 307 op/s 2024-03-11T18:35:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:55 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:35:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:55 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:35:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:55 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:35:56.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 100288 9.95 MB/sec warmup 56 sec latency 13.766 ms 2024-03-11T18:35:57.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 102902 10.07 MB/sec warmup 57 sec latency 10.854 ms 2024-03-11T18:35:57.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:57 smithi165 ceph-mon[36634]: pgmap v70: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 882 KiB/s wr, 252 op/s 2024-03-11T18:35:57.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:57 smithi023 ceph-mon[36406]: pgmap v70: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 882 KiB/s wr, 252 op/s 2024-03-11T18:35:58.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:57 smithi002 ceph-mon[31328]: pgmap v70: 129 pgs: 129 active+clean; 39 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 882 KiB/s wr, 252 op/s 2024-03-11T18:35:58.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 103912 9.94 MB/sec warmup 58 sec latency 32.647 ms 2024-03-11T18:35:59.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 106668 10.07 MB/sec warmup 59 sec latency 13.607 ms 2024-03-11T18:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 -- 172.21.15.2:0/2769410869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca0101b80 msgr2=0x7f2ca0105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 --2- 172.21.15.2:0/2769410869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca0101b80 0x7f2ca0105bc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c88009890 tx=0x7f2c8802bb70 comp rx=0 tx=0).stop 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 -- 172.21.15.2:0/2769410869 shutdown_connections 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 --2- 172.21.15.2:0/2769410869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 0x7f2ca010de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 --2- 172.21.15.2:0/2769410869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca0101b80 0x7f2ca0105bc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 --2- 172.21.15.2:0/2769410869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca01011d0 0x7f2ca01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 -- 172.21.15.2:0/2769410869 >> 172.21.15.2:0/2769410869 conn(0x7f2ca00fca30 msgr2=0x7f2ca00fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 -- 172.21.15.2:0/2769410869 shutdown_connections 2024-03-11T18:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.449+0000 7f2ca5400640 1 -- 172.21.15.2:0/2769410869 wait complete. 2024-03-11T18:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.451+0000 7f2ca5400640 1 Processor -- start 2024-03-11T18:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.451+0000 7f2ca5400640 1 -- start start 2024-03-11T18:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 0x7f2ca0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca019bea0 0x7f2ca019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca019c890 con 0x7f2ca019bea0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca019ca00 con 0x7f2ca01011d0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca5400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca019cb70 con 0x7f2ca01062a0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45526/0 (socket says 172.21.15.2:45526) 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 -- 172.21.15.2:0/725985109 learned_addr learned my addr 172.21.15.2:0/725985109 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca019bea0 0x7f2ca019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c97400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 0x7f2ca0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 -- 172.21.15.2:0/725985109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 msgr2=0x7f2ca0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 0x7f2ca0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 -- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca019bea0 msgr2=0x7f2ca019c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca019bea0 0x7f2ca019c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.452+0000 7f2c9fe00640 1 -- 172.21.15.2:0/725985109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c88009490 con 0x7f2ca01011d0 2024-03-11T18:35:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000efb0 tx=0x7f2c9000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2c9d600640 1 -- 172.21.15.2:0/725985109 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9000b570 con 0x7f2ca01011d0 2024-03-11T18:35:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2c9d600640 1 -- 172.21.15.2:0/725985109 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c9000bbb0 con 0x7f2ca01011d0 2024-03-11T18:35:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca01a4150 con 0x7f2ca01011d0 2024-03-11T18:35:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2c9d600640 1 -- 172.21.15.2:0/725985109 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90016d20 con 0x7f2ca01011d0 2024-03-11T18:35:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.453+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01a4620 con 0x7f2ca01011d0 2024-03-11T18:35:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2c9d600640 1 -- 172.21.15.2:0/725985109 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f2c9000a040 con 0x7f2ca01011d0 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 Processor -- start 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 -- start start 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 0x55b2535edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535f7570 0x55b2535f9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2535a9620 con 0x55b2535f7570 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b25355b8c0 con 0x55b2535edff0 2024-03-11T18:35:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.457+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b253558df0 con 0x55b2535eed60 2024-03-11T18:35:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45536/0 (socket says 172.21.15.2:45536) 2024-03-11T18:35:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 -- 172.21.15.2:0/3815558186 learned_addr learned my addr 172.21.15.2:0/3815558186 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 0x55b2535edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c97400640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535f7570 0x55b2535f9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 -- 172.21.15.2:0/3815558186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 msgr2=0x55b2535edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 0x55b2535edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 -- 172.21.15.2:0/3815558186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535f7570 msgr2=0x55b2535f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535f7570 0x55b2535f9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2c9fe00640 1 -- 172.21.15.2:0/3815558186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2534d5b00 con 0x55b2535edff0 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.458+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 0x55b2535edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c900cade0 tx=0x7f2c900d1ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eff0234a3d7cd9a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c9d600640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 0x7f2c840a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 0x7f2c840a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c95200640 1 -- 172.21.15.2:0/3815558186 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900966d0 con 0x55b2535edff0 2024-03-11T18:35:59.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c95200640 1 -- 172.21.15.2:0/3815558186 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c900d16b0 con 0x55b2535edff0 2024-03-11T18:35:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c9d600640 1 -- 172.21.15.2:0/725985109 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2c900cdb50 con 0x7f2ca01011d0 2024-03-11T18:35:59.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.459+0000 7f2c95200640 1 -- 172.21.15.2:0/3815558186 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9005a290 con 0x55b2535edff0 2024-03-11T18:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 -- 172.21.15.2:0/3815558186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 msgr2=0x55b2535ee470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 0x7f2c840a40c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009ef0 tx=0x7f2c98005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2c900cade0 tx=0x7f2c900d1ed0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 -- 172.21.15.2:0/3815558186 shutdown_connections 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535f7570 0x55b2535f9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535ee470 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/3815558186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535eed60 0x55b2535edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 -- 172.21.15.2:0/3815558186 >> 172.21.15.2:0/3815558186 conn(0x55b2535b54b0 msgr2=0x55b2535e8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 -- 172.21.15.2:0/3815558186 shutdown_connections 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.460+0000 7f2ca97ba740 1 -- 172.21.15.2:0/3815558186 wait complete. 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.461+0000 7f2ca97ba740 1 -- start start 2024-03-11T18:35:59.470 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.461+0000 7f2ca97ba740 1 Processor -- start 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2536018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2535fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2535545d0 con 0x55b2535eed60 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2534f9b50 con 0x55b2535f7570 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2534f97f0 con 0x55b2535edff0 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca97ba740 10 client.0 _ll_register_callbacks cb 0x55b25350d550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c97400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2536018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:60946/0 (socket says 172.21.15.2:60946) 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 -- 172.21.15.2:0/375311380 learned_addr learned my addr 172.21.15.2:0/375311380 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2535fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 -- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 msgr2=0x55b2535fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2535fb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 -- 172.21.15.2:0/375311380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 msgr2=0x55b2536018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2536018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.462+0000 7f2c9fe00640 1 -- 172.21.15.2:0/375311380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01a4620 con 0x55b2535edff0 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c9009a3c0 tx=0x7f2c900d11b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac13aa2e4dc778fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900966d0 con 0x55b2535edff0 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2c9005e7f0 con 0x55b2535edff0 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b253526cd0 con 0x55b2535edff0 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c9009a200 con 0x55b2535edff0 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2ca97ba740 10 client.25660 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b253552c50 con 0x55b2535edff0 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.463+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b253526cd0 con 0x55b2535edff0 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f2c90016410 con 0x55b2535edff0 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 10 client.25660 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 10 client.25660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 20 client.25660 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b25360b540 0x55b25360d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 4 client.25660 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b25354ace0 con 0x55b25360b540 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.464+0000 7f2c97400640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b25360b540 0x55b25360d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.465+0000 7f2c97400640 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b25360b540 0x55b25360d930 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d35e5d0de2e87959 server_cookie=9bd20db887bfcc35 in_seq=0 out_seq=0 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.465+0000 7f2c8d600640 10 client.25660 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2c8d600640 1 -- 172.21.15.2:0/375311380 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b25354ace0 con 0x55b25360b540 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2c8d600640 10 client.25660 handle_command_reply: tid=0 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 client.25660 shutdown 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 msgr2=0x55b2536013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c9009a3c0 tx=0x7f2c900d11b0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 shutdown_connections 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b25360b540 0x55b25360d930 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2535fb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2536018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2c8d600640 0 client.25660 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/375311380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535edff0 0x55b2536013b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.466+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 >> 172.21.15.2:0/375311380 conn(0x55b2535b54b0 msgr2=0x55b2535eaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.467+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 shutdown_connections 2024-03-11T18:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.467+0000 7f2ca97ba740 1 -- 172.21.15.2:0/375311380 wait complete. 2024-03-11T18:35:59.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.467+0000 7f2ca97ba740 20 client.25660 trim_cache size 0 max 16384 2024-03-11T18:35:59.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 Processor -- start 2024-03-11T18:35:59.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 -- start start 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2535e8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b25329ce60 con 0x55b2535eed60 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b253526d00 con 0x55b2535edff0 2024-03-11T18:35:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2535ef350 con 0x55b2535f7570 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca4a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2c9fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45564/0 (socket says 172.21.15.2:45564) 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2c97400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2535e8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2ca4a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:60958/0 (socket says 172.21.15.2:60958) 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.482+0000 7f2c9fe00640 1 -- 172.21.15.2:0/827840698 learned_addr learned my addr 172.21.15.2:0/827840698 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 -- 172.21.15.2:0/827840698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 msgr2=0x55b2535fb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 -- 172.21.15.2:0/827840698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 msgr2=0x55b2535e8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2535e8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 -- 172.21.15.2:0/827840698 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2534ef530 con 0x55b2535edff0 2024-03-11T18:35:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2c900d1220 tx=0x7f2c90098940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9250f4f1b8f38a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c8fe00640 1 -- 172.21.15.2:0/827840698 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c900966d0 con 0x55b2535edff0 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c8fe00640 1 -- 172.21.15.2:0/827840698 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c90099f70 con 0x55b2535edff0 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.483+0000 7f2c8fe00640 1 -- 172.21.15.2:0/827840698 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90016410 con 0x55b2535edff0 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 -- 172.21.15.2:0/827840698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 msgr2=0x55b2535e8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2c900d1220 tx=0x7f2c90098940 comp rx=0 tx=0).stop 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 -- 172.21.15.2:0/827840698 shutdown_connections 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535f7570 0x55b2535fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b2535e8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/827840698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535edff0 0x55b2535e8810 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 -- 172.21.15.2:0/827840698 >> 172.21.15.2:0/827840698 conn(0x55b2535b54b0 msgr2=0x55b2536122a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 -- 172.21.15.2:0/827840698 shutdown_connections 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.484+0000 7f2ca97ba740 1 -- 172.21.15.2:0/827840698 wait complete. 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 -- start start 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 Processor -- start 2024-03-11T18:35:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b253601070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535ff4c0 0x55b2535ff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2535fff00 con 0x55b2535eed60 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b253600070 con 0x55b2535f7570 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b253600230 con 0x55b2535ff4c0 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2ca97ba740 10 client.0 _ll_register_callbacks cb 0x7f2c90009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:35:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.485+0000 7f2c97400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45574/0 (socket says 172.21.15.2:45574) 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 -- 172.21.15.2:0/1923966565 learned_addr learned my addr 172.21.15.2:0/1923966565 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2ca4a00640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535ff4c0 0x55b2535ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b253601070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 -- 172.21.15.2:0/1923966565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535ff4c0 msgr2=0x55b2535ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535ff4c0 0x55b2535ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 -- 172.21.15.2:0/1923966565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 msgr2=0x55b253601070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b253601070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 -- 172.21.15.2:0/1923966565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98009830 con 0x55b2535f7570 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c97400640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c8802bbb0 tx=0x7f2c88000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d850a45b4b6dc353 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.486+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c88044050 con 0x55b2535f7570 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2c88046e40 con 0x55b2535f7570 2024-03-11T18:35:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b253579d30 con 0x55b2535f7570 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c88004240 con 0x55b2535f7570 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 10 client.25667 fetch_fsmap learned FSMap version 26 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b253552c50 con 0x55b2535f7570 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b253579d30 con 0x55b2535f7570 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f2c8802c940 con 0x55b2535f7570 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 10 client.25667 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 10 client.25667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 20 client.25667 populate_metadata read hostname 'smithi002' 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b253662fd0 0x55b2536653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:35:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 4 client.25667 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:35:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.487+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b253254600 con 0x55b253662fd0 2024-03-11T18:35:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.488+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b253662fd0 0x55b2536653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:35:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.488+0000 7f2c9fe00640 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b253662fd0 0x55b2536653f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c04628177ae74fa server_cookie=7d6917608758ec47 in_seq=0 out_seq=0 2024-03-11T18:35:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.488+0000 7f2c8d600640 10 client.25667 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.716+0000 7f2c8d600640 1 -- 172.21.15.2:0/1923966565 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b253254600 con 0x55b253662fd0 2024-03-11T18:35:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.716+0000 7f2c8d600640 10 client.25667 handle_command_reply: tid=0 2024-03-11T18:35:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.716+0000 7f2ca97ba740 1 client.25667 shutdown 2024-03-11T18:35:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.716+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 msgr2=0x55b2536015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.716+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2c8802bbb0 tx=0x7f2c88000bc0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 shutdown_connections 2024-03-11T18:35:59.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b253662fd0 0x55b2536653f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b2535ff4c0 0x55b2535ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b2535f7570 0x55b2536015b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 --2- 172.21.15.2:0/1923966565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b2535eed60 0x55b253601070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2c8d600640 0 client.25667 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 >> 172.21.15.2:0/1923966565 conn(0x55b2535b54b0 msgr2=0x55b2535e8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 shutdown_connections 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 1 -- 172.21.15.2:0/1923966565 wait complete. 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.717+0000 7f2ca97ba740 20 client.25667 trim_cache size 0 max 16384 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:35:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 msgr2=0x7f2c840a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 0x7f2c840a40c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c98009ef0 tx=0x7f2c98005f90 comp rx=0 tx=0).stop 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 msgr2=0x7f2ca01975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2c9000efb0 tx=0x7f2c9000c800 comp rx=0 tx=0).stop 2024-03-11T18:35:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 shutdown_connections 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2c840a1c00 0x7f2c840a40c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ca019bea0 0x7f2ca019c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ca01062a0 0x7f2ca0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 --2- 172.21.15.2:0/725985109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ca01011d0 0x7f2ca01975e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 >> 172.21.15.2:0/725985109 conn(0x7f2ca00fca30 msgr2=0x7f2ca00fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 shutdown_connections 2024-03-11T18:35:59.722 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:35:59.718+0000 7f2ca5400640 1 -- 172.21.15.2:0/725985109 wait complete. 2024-03-11T18:35:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:35:59 smithi165 ceph-mon[36634]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-11T18:35:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:35:59 smithi023 ceph-mon[36406]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-11T18:36:00.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:35:59 smithi002 ceph-mon[31328]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 324 op/s 2024-03-11T18:36:00.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 107875 10.00 MB/sec warmup 60 sec latency 79.048 ms 2024-03-11T18:36:01.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 108738 9.86 MB/sec warmup 61 sec latency 86.608 ms 2024-03-11T18:36:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:01 smithi165 ceph-mon[36634]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:01 smithi023 ceph-mon[36406]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:01 smithi002 ceph-mon[31328]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:02.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 109995 9.87 MB/sec warmup 62 sec latency 84.192 ms 2024-03-11T18:36:03.382 INFO:tasks.workunit.client.0.smithi002.stdout: 1 110993 9.76 MB/sec warmup 63 sec latency 51.064 ms 2024-03-11T18:36:03.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:03 smithi165 ceph-mon[36634]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:03.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:03 smithi023 ceph-mon[36406]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:04.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:03 smithi002 ceph-mon[31328]: pgmap v73: 129 pgs: 129 active+clean; 46 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-03-11T18:36:04.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 113630 9.87 MB/sec warmup 64 sec latency 20.653 ms 2024-03-11T18:36:05.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 114928 9.81 MB/sec warmup 65 sec latency 12.186 ms 2024-03-11T18:36:05.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:05 smithi023 ceph-mon[36406]: pgmap v74: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-11T18:36:05.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:05 smithi165 ceph-mon[36634]: pgmap v74: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-11T18:36:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:05 smithi002 ceph-mon[31328]: pgmap v74: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 360 op/s 2024-03-11T18:36:06.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 117358 9.87 MB/sec warmup 66 sec latency 9.020 ms 2024-03-11T18:36:07.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 118730 9.81 MB/sec warmup 67 sec latency 15.456 ms 2024-03-11T18:36:07.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:07 smithi165 ceph-mon[36634]: pgmap v75: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-11T18:36:07.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:07 smithi023 ceph-mon[36406]: pgmap v75: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-11T18:36:08.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:07 smithi002 ceph-mon[31328]: pgmap v75: 129 pgs: 129 active+clean; 44 MiB data, 668 MiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-11T18:36:08.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 120917 9.86 MB/sec warmup 68 sec latency 57.361 ms 2024-03-11T18:36:09.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 122273 9.80 MB/sec warmup 69 sec latency 16.733 ms 2024-03-11T18:36:09.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:09 smithi165 ceph-mon[36634]: pgmap v76: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 391 op/s 2024-03-11T18:36:09.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:09 smithi023 ceph-mon[36406]: pgmap v76: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 391 op/s 2024-03-11T18:36:10.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:09 smithi002 ceph-mon[31328]: pgmap v76: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 391 op/s 2024-03-11T18:36:10.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 124565 9.86 MB/sec warmup 70 sec latency 11.088 ms 2024-03-11T18:36:11.383 INFO:tasks.workunit.client.0.smithi002.stdout: 1 125950 9.80 MB/sec warmup 71 sec latency 15.376 ms 2024-03-11T18:36:11.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:11 smithi165 ceph-mon[36634]: pgmap v77: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:11.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:11 smithi023 ceph-mon[36406]: pgmap v77: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:12.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:11 smithi002 ceph-mon[31328]: pgmap v77: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:12.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 127976 9.85 MB/sec warmup 72 sec latency 11.904 ms 2024-03-11T18:36:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 -- 172.21.15.2:0/17964304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c1062a0 msgr2=0x7f043c10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 --2- 172.21.15.2:0/17964304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c1062a0 0x7f043c10de70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0430009890 tx=0x7f043002bb70 comp rx=0 tx=0).stop 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 -- 172.21.15.2:0/17964304 shutdown_connections 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 --2- 172.21.15.2:0/17964304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c1062a0 0x7f043c10de70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 --2- 172.21.15.2:0/17964304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c101b80 0x7f043c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 --2- 172.21.15.2:0/17964304 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 -- 172.21.15.2:0/17964304 >> 172.21.15.2:0/17964304 conn(0x7f043c0fca30 msgr2=0x7f043c0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 -- 172.21.15.2:0/17964304 shutdown_connections 2024-03-11T18:36:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.867+0000 7f0442800640 1 -- 172.21.15.2:0/17964304 wait complete. 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 Processor -- start 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 -- start start 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c1062a0 0x7f043c19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 0x7f043c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c19a140 con 0x7f043c1062a0 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c19a2b0 con 0x7f043c199670 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0442800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c19a470 con 0x7f043c1011d0 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0441400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0441400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39220/0 (socket says 172.21.15.2:39220) 2024-03-11T18:36:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.869+0000 7f0441400640 1 -- 172.21.15.2:0/4201174244 learned_addr learned my addr 172.21.15.2:0/4201174244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441e00640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 0x7f043c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0440a00640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c1062a0 0x7f043c19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 -- 172.21.15.2:0/4201174244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 msgr2=0x7f043c199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 0x7f043c199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 -- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c1062a0 msgr2=0x7f043c19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c1062a0 0x7f043c19faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 -- 172.21.15.2:0/4201174244 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0430009490 con 0x7f043c1011d0 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441e00640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 0x7f043c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:36:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.870+0000 7f0441400640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f043400ed80 tx=0x7f043400c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.871+0000 7f043a000640 1 -- 172.21.15.2:0/4201174244 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0434017640 con 0x7f043c1011d0 2024-03-11T18:36:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.871+0000 7f043a000640 1 -- 172.21.15.2:0/4201174244 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0434017c80 con 0x7f043c1011d0 2024-03-11T18:36:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.871+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c1a6410 con 0x7f043c1011d0 2024-03-11T18:36:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.871+0000 7f043a000640 1 -- 172.21.15.2:0/4201174244 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043400fda0 con 0x7f043c1011d0 2024-03-11T18:36:12.874 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.871+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c1a6940 con 0x7f043c1011d0 2024-03-11T18:36:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.874+0000 7f0446bd2740 1 Processor -- start 2024-03-11T18:36:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.874+0000 7f0446bd2740 1 -- start start 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 0x561b440cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4407f620 con 0x561b440cd570 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440318c0 con 0x561b440c4d60 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0446bd2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4402edf0 con 0x561b440c3ff0 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f043a000640 1 -- 172.21.15.2:0/4201174244 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f043400a040 con 0x7f043c1011d0 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0441e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0441400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0441e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50842/0 (socket says 172.21.15.2:50842) 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.875+0000 7f0441e00640 1 -- 172.21.15.2:0/1431902707 learned_addr learned my addr 172.21.15.2:0/1431902707 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0440a00640 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 0x561b440cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 -- 172.21.15.2:0/1431902707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 msgr2=0x561b440c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 -- 172.21.15.2:0/1431902707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 msgr2=0x561b440cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 0x561b440cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 -- 172.21.15.2:0/1431902707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b43fabb00 con 0x561b440c4d60 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0440a00640 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 0x561b440cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:36:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.876+0000 7f0441e00640 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0430009890 tx=0x7f04300056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d56b989dc0d9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f042e600640 1 -- 172.21.15.2:0/1431902707 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04300042c0 con 0x561b440c4d60 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f042e600640 1 -- 172.21.15.2:0/1431902707 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0430004420 con 0x561b440c4d60 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f042e600640 1 -- 172.21.15.2:0/1431902707 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04300035f0 con 0x561b440c4d60 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f043a000640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 0x7f04240a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f043a000640 1 -- 172.21.15.2:0/4201174244 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f04340cdac0 con 0x7f043c1011d0 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f0446bd2740 1 -- 172.21.15.2:0/1431902707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 msgr2=0x561b440c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0430009890 tx=0x7f04300056d0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.877+0000 7f0441400640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 0x7f04240a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 -- 172.21.15.2:0/1431902707 shutdown_connections 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440cd570 0x561b440cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1431902707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c3ab0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 -- 172.21.15.2:0/1431902707 >> 172.21.15.2:0/1431902707 conn(0x561b4408b4b0 msgr2=0x561b440bece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 -- 172.21.15.2:0/1431902707 shutdown_connections 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 -- 172.21.15.2:0/1431902707 wait complete. 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0441400640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 0x7f04240a40c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x561b440c54c0 tx=0x7f04340d1480 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.878+0000 7f0446bd2740 1 -- start start 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 Processor -- start 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c3ff0 0x561b440ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440cd570 0x561b440cb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4402a5d0 con 0x561b440c3ff0 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b43fcfb50 con 0x561b440cd570 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b43fcf7f0 con 0x561b440c4d60 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0446bd2740 10 client.0 _ll_register_callbacks cb 0x561b43fe3550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0440a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0441e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c3ff0 0x561b440ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0440a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39240/0 (socket says 172.21.15.2:39240) 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.879+0000 7f0440a00640 1 -- 172.21.15.2:0/1495079730 learned_addr learned my addr 172.21.15.2:0/1495079730 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0441e00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440cd570 0x561b440cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 -- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440cd570 msgr2=0x561b440cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440cd570 0x561b440cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 -- 172.21.15.2:0/1495079730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c3ff0 msgr2=0x561b440ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c3ff0 0x561b440ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 -- 172.21.15.2:0/1495079730 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c10e680 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0440a00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f042800bdf0 tx=0x7f04280117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6653270a2106e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0422a00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.880+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428012860 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b43ffccd0 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f0428018070 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0428011ec0 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0446bd2740 10 client.25678 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561b44028c50 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561b43ffccd0 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f042801dcd0 con 0x561b440c4d60 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0446bd2740 10 client.25678 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.881+0000 7f0446bd2740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0446bd2740 20 client.25678 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440dd460 0x561b440df880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0446bd2740 4 client.25678 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b44020ce0 con 0x561b440dd460 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0441e00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440dd460 0x561b440df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0441e00640 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440dd460 0x561b440df880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2ae983d7e86a8e server_cookie=e0531e0860687e8c in_seq=0 out_seq=0 2024-03-11T18:36:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.882+0000 7f0422a00640 10 client.25678 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.883+0000 7f0422a00640 1 -- 172.21.15.2:0/1495079730 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b44020ce0 con 0x561b440dd460 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.883+0000 7f0422a00640 10 client.25678 handle_command_reply: tid=0 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.883+0000 7f0446bd2740 1 client.25678 shutdown 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.883+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 msgr2=0x561b440cab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.883+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f042800bdf0 tx=0x7f04280117b0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 shutdown_connections 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440dd460 0x561b440df880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440cd570 0x561b440cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c4d60 0x561b440cab10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0422a00640 0 client.25678 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1495079730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c3ff0 0x561b440ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 >> 172.21.15.2:0/1495079730 conn(0x561b4408b4b0 msgr2=0x561b440c0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 shutdown_connections 2024-03-11T18:36:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 1 -- 172.21.15.2:0/1495079730 wait complete. 2024-03-11T18:36:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.884+0000 7f0446bd2740 20 client.25678 trim_cache size 0 max 16384 2024-03-11T18:36:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 Processor -- start 2024-03-11T18:36:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 -- start start 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 0x561b440ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b43d72e60 con 0x561b4408b4b0 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440cb920 con 0x561b440c4d60 2024-03-11T18:36:12.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.902+0000 7f0446bd2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440bfd10 con 0x561b440c3ff0 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39254/0 (socket says 172.21.15.2:39254) 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 -- 172.21.15.2:0/3350678699 learned_addr learned my addr 172.21.15.2:0/3350678699 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0441e00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 0x561b440ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0441e00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 -- 172.21.15.2:0/3350678699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 msgr2=0x561b440c8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 -- 172.21.15.2:0/3350678699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 msgr2=0x561b440ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 0x561b440ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 -- 172.21.15.2:0/3350678699 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b43fc5530 con 0x561b440c3ff0 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0441e00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 0x561b440ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:36:12.906 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f0440a00640 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f042800c010 tx=0x7f042800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38101cfbf37e11a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.903+0000 7f042dc00640 1 -- 172.21.15.2:0/3350678699 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042800f810 con 0x561b440c3ff0 2024-03-11T18:36:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.904+0000 7f042dc00640 1 -- 172.21.15.2:0/3350678699 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0428016070 con 0x561b440c3ff0 2024-03-11T18:36:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.904+0000 7f042dc00640 1 -- 172.21.15.2:0/3350678699 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428012660 con 0x561b440c3ff0 2024-03-11T18:36:12.907 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.904+0000 7f0446bd2740 1 -- 172.21.15.2:0/3350678699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 msgr2=0x561b440cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.904+0000 7f0446bd2740 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f042800c010 tx=0x7f042800b8f0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 -- 172.21.15.2:0/3350678699 shutdown_connections 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b440c4d60 0x561b440c8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440cae70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 --2- 172.21.15.2:0/3350678699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b4408b4b0 0x561b440ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 -- 172.21.15.2:0/3350678699 >> 172.21.15.2:0/3350678699 conn(0x561b440ba170 msgr2=0x561b440c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 -- 172.21.15.2:0/3350678699 shutdown_connections 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 -- 172.21.15.2:0/3350678699 wait complete. 2024-03-11T18:36:12.908 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.905+0000 7f0446bd2740 1 -- start start 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 Processor -- start 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440d4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c4d60 0x561b440d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440d2ef0 con 0x561b440c4d60 2024-03-11T18:36:12.909 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440d3060 con 0x561b4408b4b0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b440d3220 con 0x561b440c3ff0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.906+0000 7f0446bd2740 10 client.0 _ll_register_callbacks cb 0x7f042800ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50874/0 (socket says 172.21.15.2:50874) 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 -- 172.21.15.2:0/1852560910 learned_addr learned my addr 172.21.15.2:0/1852560910 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c4d60 0x561b440d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 -- 172.21.15.2:0/1852560910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 msgr2=0x561b440d4720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440d4720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 -- 172.21.15.2:0/1852560910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c4d60 msgr2=0x561b440d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c4d60 0x561b440d27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 -- 172.21.15.2:0/1852560910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b44028c50 con 0x561b4408b4b0 2024-03-11T18:36:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0441e00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f043002f020 tx=0x7f043002c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ee9e2a4fe9850d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.907+0000 7f0422a00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043005d070 con 0x561b4408b4b0 2024-03-11T18:36:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f04300573a0 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b4404fd30 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0430004470 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0446bd2740 10 client.25685 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.908+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x561b43feb2c0 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561b4404fd30 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0430055070 con 0x561b4408b4b0 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 10 client.25685 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 10 client.25685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 20 client.25685 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440cd570 0x561b4413f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 4 client.25685 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:36:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561b43d2a600 con 0x561b440cd570 2024-03-11T18:36:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.909+0000 7f0440a00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440cd570 0x561b4413f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.910+0000 7f0440a00640 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440cd570 0x561b4413f0f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c5a16f04bc811c1 server_cookie=73cc47ba81c2ea6f in_seq=0 out_seq=0 2024-03-11T18:36:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:12.910+0000 7f0422a00640 10 client.25685 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.163+0000 7f0422a00640 1 -- 172.21.15.2:0/1852560910 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561b43d2a600 con 0x561b440cd570 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.163+0000 7f0422a00640 10 client.25685 handle_command_reply: tid=0 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.163+0000 7f0446bd2740 1 client.25685 shutdown 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 msgr2=0x561b440d41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f043002f020 tx=0x7f043002c860 comp rx=0 tx=0).stop 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 shutdown_connections 2024-03-11T18:36:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561b440cd570 0x561b4413f0f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561b440c4d60 0x561b440d27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561b440c3ff0 0x561b440d4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0422a00640 0 client.25685 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 --2- 172.21.15.2:0/1852560910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561b4408b4b0 0x561b440d41e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 >> 172.21.15.2:0/1852560910 conn(0x561b440ba170 msgr2=0x561b440c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 shutdown_connections 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 1 -- 172.21.15.2:0/1852560910 wait complete. 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.164+0000 7f0446bd2740 20 client.25685 trim_cache size 0 max 16384 2024-03-11T18:36:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 msgr2=0x7f04240a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 0x7f04240a40c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x561b440c54c0 tx=0x7f04340d1480 comp rx=0 tx=0).stop 2024-03-11T18:36:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 msgr2=0x7f043c19f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f043400ed80 tx=0x7f043400c6e0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 shutdown_connections 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f043c199670 0x7f043c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04240a1c00 0x7f04240a40c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f043c1062a0 0x7f043c19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 --2- 172.21.15.2:0/4201174244 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f043c1011d0 0x7f043c19f5b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.165+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 >> 172.21.15.2:0/4201174244 conn(0x7f043c0fca30 msgr2=0x7f043c10c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.166+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 shutdown_connections 2024-03-11T18:36:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:13.166+0000 7f0442800640 1 -- 172.21.15.2:0/4201174244 wait complete. 2024-03-11T18:36:13.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 128863 9.75 MB/sec warmup 73 sec latency 160.618 ms 2024-03-11T18:36:13.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:13 smithi023 ceph-mon[36406]: pgmap v78: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:13.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:13 smithi165 ceph-mon[36634]: pgmap v78: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:14.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:13 smithi002 ceph-mon[31328]: pgmap v78: 129 pgs: 129 active+clean; 48 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.0 MiB/s wr, 319 op/s 2024-03-11T18:36:14.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 131096 9.78 MB/sec warmup 74 sec latency 31.748 ms 2024-03-11T18:36:15.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 132163 9.72 MB/sec warmup 75 sec latency 16.196 ms 2024-03-11T18:36:15.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:15 smithi023 ceph-mon[36406]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 505 op/s 2024-03-11T18:36:15.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:15 smithi165 ceph-mon[36634]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 505 op/s 2024-03-11T18:36:16.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:15 smithi002 ceph-mon[31328]: pgmap v79: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 505 op/s 2024-03-11T18:36:16.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 134274 9.75 MB/sec warmup 76 sec latency 18.443 ms 2024-03-11T18:36:17.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 135531 9.71 MB/sec warmup 77 sec latency 17.647 ms 2024-03-11T18:36:17.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:17 smithi023 ceph-mon[36406]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-03-11T18:36:17.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:17 smithi165 ceph-mon[36634]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-03-11T18:36:18.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:17 smithi002 ceph-mon[31328]: pgmap v80: 129 pgs: 129 active+clean; 44 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-03-11T18:36:18.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 137238 9.68 MB/sec warmup 78 sec latency 68.326 ms 2024-03-11T18:36:19.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 138923 9.71 MB/sec warmup 79 sec latency 22.285 ms 2024-03-11T18:36:19.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:19 smithi023 ceph-mon[36406]: pgmap v81: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 377 op/s 2024-03-11T18:36:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:19 smithi165 ceph-mon[36634]: pgmap v81: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 377 op/s 2024-03-11T18:36:20.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:19 smithi002 ceph-mon[31328]: pgmap v81: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 377 op/s 2024-03-11T18:36:20.384 INFO:tasks.workunit.client.0.smithi002.stdout: 1 140752 9.67 MB/sec warmup 80 sec latency 17.751 ms 2024-03-11T18:36:21.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 142368 9.70 MB/sec warmup 81 sec latency 14.909 ms 2024-03-11T18:36:21.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:21 smithi165 ceph-mon[36634]: pgmap v82: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:21.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:21 smithi023 ceph-mon[36406]: pgmap v82: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:22.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:21 smithi002 ceph-mon[31328]: pgmap v82: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:22.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 143828 9.65 MB/sec warmup 82 sec latency 34.948 ms 2024-03-11T18:36:23.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 145741 9.69 MB/sec warmup 83 sec latency 65.821 ms 2024-03-11T18:36:23.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:23 smithi165 ceph-mon[36634]: pgmap v83: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:23.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:23 smithi023 ceph-mon[36406]: pgmap v83: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:24.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:23 smithi002 ceph-mon[31328]: pgmap v83: 129 pgs: 129 active+clean; 48 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:36:24.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 146710 9.61 MB/sec warmup 84 sec latency 22.612 ms 2024-03-11T18:36:25.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 149142 9.68 MB/sec warmup 85 sec latency 12.311 ms 2024-03-11T18:36:25.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:25 smithi165 ceph-mon[36634]: pgmap v84: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-11T18:36:25.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:25 smithi023 ceph-mon[36406]: pgmap v84: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-11T18:36:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:25 smithi002 ceph-mon[31328]: pgmap v84: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 369 op/s 2024-03-11T18:36:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 -- 172.21.15.2:0/3803431340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78106460 msgr2=0x7f2e7810e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 --2- 172.21.15.2:0/3803431340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78106460 0x7f2e7810e030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009c10 tx=0x7f2e7002bef0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 -- 172.21.15.2:0/3803431340 shutdown_connections 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 --2- 172.21.15.2:0/3803431340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78106460 0x7f2e7810e030 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 --2- 172.21.15.2:0/3803431340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78101d40 0x7f2e78105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 --2- 172.21.15.2:0/3803431340 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78101390 0x7f2e78101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.328+0000 7f2e7e800640 1 -- 172.21.15.2:0/3803431340 >> 172.21.15.2:0/3803431340 conn(0x7f2e780fcbd0 msgr2=0x7f2e780feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.329+0000 7f2e7e800640 1 -- 172.21.15.2:0/3803431340 shutdown_connections 2024-03-11T18:36:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.329+0000 7f2e7e800640 1 -- 172.21.15.2:0/3803431340 wait complete. 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.329+0000 7f2e7e800640 1 Processor -- start 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 -- start start 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 0x7f2e780739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78073f00 0x7f2e781a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781a6020 con 0x7f2e78073f00 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781a6190 con 0x7f2e78101390 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e7e800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e781a6350 con 0x7f2e78106460 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e77400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78073f00 0x7f2e781a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 0x7f2e780739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.330+0000 7f2e77400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46988/0 (socket says 172.21.15.2:46988) 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 -- 172.21.15.2:0/38014365 learned_addr learned my addr 172.21.15.2:0/38014365 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 -- 172.21.15.2:0/38014365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 msgr2=0x7f2e780739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 0x7f2e780739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 -- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78073f00 msgr2=0x7f2e781a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78073f00 0x7f2e781a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 -- 172.21.15.2:0/38014365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e70009840 con 0x7f2e78101390 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 0x7f2e780739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:36:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e77400640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800e9f0 tx=0x7f2e6800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.331+0000 7f2e74c00640 1 -- 172.21.15.2:0/38014365 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68009720 con 0x7f2e78101390 2024-03-11T18:36:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.332+0000 7f2e74c00640 1 -- 172.21.15.2:0/38014365 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e68005500 con 0x7f2e78101390 2024-03-11T18:36:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.332+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e781a6630 con 0x7f2e78101390 2024-03-11T18:36:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.332+0000 7f2e74c00640 1 -- 172.21.15.2:0/38014365 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e68010430 con 0x7f2e78101390 2024-03-11T18:36:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.332+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e781a6af0 con 0x7f2e78101390 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.335+0000 7f2e8156c740 1 Processor -- start 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.335+0000 7f2e8156c740 1 -- start start 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.335+0000 7f2e74c00640 1 -- 172.21.15.2:0/38014365 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f2e68005020 con 0x7f2e78101390 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.335+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.335+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640be230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e8156c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563964079620 con 0x5639640beb20 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e8156c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396402b8c0 con 0x5639640c7330 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e8156c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563964028df0 con 0x5639640bddb0 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46990/0 (socket says 172.21.15.2:46990) 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 -- 172.21.15.2:0/3666660941 learned_addr learned my addr 172.21.15.2:0/3666660941 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e77400640 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 -- 172.21.15.2:0/3666660941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 msgr2=0x5639640be230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640be230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 -- 172.21.15.2:0/3666660941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 msgr2=0x5639640bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.336+0000 7f2e6fe00640 1 -- 172.21.15.2:0/3666660941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563963fa5b00 con 0x5639640c7330 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2e60011470 tx=0x7f2e60011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=284e831cd3ca2861 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e74c00640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 0x7f2e5c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e6cc00640 1 -- 172.21.15.2:0/3666660941 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000f600 con 0x5639640c7330 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e6cc00640 1 -- 172.21.15.2:0/3666660941 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e6000fc40 con 0x5639640c7330 2024-03-11T18:36:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e6cc00640 1 -- 172.21.15.2:0/3666660941 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60019da0 con 0x5639640c7330 2024-03-11T18:36:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e74c00640 1 -- 172.21.15.2:0/38014365 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2e68014070 con 0x7f2e78101390 2024-03-11T18:36:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.337+0000 7f2e77e00640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 0x7f2e5c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 -- 172.21.15.2:0/3666660941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 msgr2=0x5639640c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2e60011470 tx=0x7f2e60011930 comp rx=0 tx=0).stop 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 -- 172.21.15.2:0/3666660941 shutdown_connections 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c9720 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 --2- 172.21.15.2:0/3666660941 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 -- 172.21.15.2:0/3666660941 >> 172.21.15.2:0/3666660941 conn(0x5639640854b0 msgr2=0x5639640b8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 -- 172.21.15.2:0/3666660941 shutdown_connections 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e77e00640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 0x7f2e5c0a4050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2e70004af0 tx=0x7f2e70005fd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.338+0000 7f2e8156c740 1 -- 172.21.15.2:0/3666660941 wait complete. 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 -- start start 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 Processor -- start 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640c7330 0x5639640c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639640245d0 con 0x5639640bddb0 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563963fc9b50 con 0x5639640beb20 2024-03-11T18:36:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.339+0000 7f2e8156c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563963fc97f0 con 0x5639640c7330 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e8156c740 10 client.0 _ll_register_callbacks cb 0x563963fdd550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e77400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46994/0 (socket says 172.21.15.2:46994) 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 -- 172.21.15.2:0/2942415283 learned_addr learned my addr 172.21.15.2:0/2942415283 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e77e00640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640c7330 0x5639640c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 -- 172.21.15.2:0/2942415283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640c7330 msgr2=0x5639640c4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640c7330 0x5639640c4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 -- 172.21.15.2:0/2942415283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 msgr2=0x5639640c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 -- 172.21.15.2:0/2942415283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e60009c40 con 0x5639640beb20 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006010 tx=0x7f2e60019860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e428076a6f71ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.340+0000 7f2e65600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6001aea0 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563963ff6cd0 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2e60005250 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e60005410 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e8156c740 10 client.25697 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563964022c50 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.341+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563963ff6cd0 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f2e6000f900 con 0x5639640beb20 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 10 client.25697 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 20 client.25697 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5639640d7270 0x5639640d96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 4 client.25697 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:36:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56396401ace0 con 0x5639640d7270 2024-03-11T18:36:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.342+0000 7f2e77400640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5639640d7270 0x5639640d96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.343+0000 7f2e77400640 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5639640d7270 0x5639640d96b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0d6bf53c1feb0fd server_cookie=a7f62b6b8563490d in_seq=0 out_seq=0 2024-03-11T18:36:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.343+0000 7f2e65600640 10 client.25697 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.343+0000 7f2e65600640 1 -- 172.21.15.2:0/2942415283 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56396401ace0 con 0x5639640d7270 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.343+0000 7f2e65600640 10 client.25697 handle_command_reply: tid=0 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.343+0000 7f2e8156c740 1 client.25697 shutdown 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 msgr2=0x5639640c48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2e60006010 tx=0x7f2e60019860 comp rx=0 tx=0).stop 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 shutdown_connections 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5639640d7270 0x5639640d96b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.350 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640c7330 0x5639640c4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640beb20 0x5639640c48b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e65600640 0 client.25697 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 --2- 172.21.15.2:0/2942415283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 >> 172.21.15.2:0/2942415283 conn(0x5639640854b0 msgr2=0x5639640ba7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 shutdown_connections 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 1 -- 172.21.15.2:0/2942415283 wait complete. 2024-03-11T18:36:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.344+0000 7f2e8156c740 20 client.25697 trim_cache size 0 max 16384 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.359+0000 7f2e8156c740 1 Processor -- start 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 -- start start 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563963d6ce60 con 0x5639640bddb0 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639640b98a0 con 0x5639640c7330 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e8156c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639640bef00 con 0x5639640beb20 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e77400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:44486/0 (socket says 172.21.15.2:44486) 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e77400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54242/0 (socket says 172.21.15.2:54242) 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e6fe00640 1 -- 172.21.15.2:0/4075843007 learned_addr learned my addr 172.21.15.2:0/4075843007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:26.364 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.360+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46996/0 (socket says 172.21.15.2:46996) 2024-03-11T18:36:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 -- 172.21.15.2:0/4075843007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 msgr2=0x5639640c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 -- 172.21.15.2:0/4075843007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 msgr2=0x5639640c6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 -- 172.21.15.2:0/4075843007 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563963fbf530 con 0x5639640beb20 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e6fe00640 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009f30 tx=0x7f2e60011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ddaa181a1d9acdc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e67e00640 1 -- 172.21.15.2:0/4075843007 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60011230 con 0x5639640beb20 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e67e00640 1 -- 172.21.15.2:0/4075843007 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e6000f540 con 0x5639640beb20 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.361+0000 7f2e67e00640 1 -- 172.21.15.2:0/4075843007 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60019c00 con 0x5639640beb20 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 -- 172.21.15.2:0/4075843007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 msgr2=0x5639640c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2e60009f30 tx=0x7f2e60011930 comp rx=0 tx=0).stop 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 -- 172.21.15.2:0/4075843007 shutdown_connections 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640beb20 0x5639640c4c10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 --2- 172.21.15.2:0/4075843007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640bddb0 0x5639640c6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 -- 172.21.15.2:0/4075843007 >> 172.21.15.2:0/4075843007 conn(0x5639640854b0 msgr2=0x5639640c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 -- 172.21.15.2:0/4075843007 shutdown_connections 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.362+0000 7f2e8156c740 1 -- 172.21.15.2:0/4075843007 wait complete. 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 -- start start 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 Processor -- start 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396412d910 con 0x5639640beb20 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396412da80 con 0x5639640c7330 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56396412dc40 con 0x5639640bddb0 2024-03-11T18:36:26.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.363+0000 7f2e8156c740 10 client.0 _ll_register_callbacks cb 0x7f2e6001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e6fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:47004/0 (socket says 172.21.15.2:47004) 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:44502/0 (socket says 172.21.15.2:44502) 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 -- 172.21.15.2:0/1228128396 learned_addr learned my addr 172.21.15.2:0/1228128396 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 -- 172.21.15.2:0/1228128396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 msgr2=0x5639640cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 -- 172.21.15.2:0/1228128396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 msgr2=0x5639640cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 -- 172.21.15.2:0/1228128396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563964022c50 con 0x5639640c7330 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77400640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:36:26.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e77e00640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e70035000 tx=0x7f2e70009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1f7f724b52db75 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.364+0000 7f2e65600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70004740 con 0x5639640c7330 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563964049d30 con 0x5639640c7330 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f2e70049450 con 0x5639640c7330 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e700048a0 con 0x5639640c7330 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e8156c740 10 client.25703 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563963fe52c0 con 0x5639640c7330 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563964049d30 con 0x5639640c7330 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.365+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f2e70034730 con 0x5639640c7330 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 10 client.25703 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 10 client.25703 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 20 client.25703 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56396412f500 0x563964131920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 4 client.25703 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:36:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563963d24600 con 0x56396412f500 2024-03-11T18:36:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e77400640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56396412f500 0x563964131920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e77400640 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56396412f500 0x563964131920 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaad78a21ae28117 server_cookie=2077e0d8bab66caf in_seq=0 out_seq=0 2024-03-11T18:36:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.366+0000 7f2e65600640 10 client.25703 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:26.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 150258 9.61 MB/sec warmup 86 sec latency 20.954 ms 2024-03-11T18:36:26.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e65600640 1 -- 172.21.15.2:0/1228128396 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563963d24600 con 0x56396412f500 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e65600640 10 client.25703 handle_command_reply: tid=0 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 client.25703 shutdown 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 msgr2=0x5639640cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e70035000 tx=0x7f2e70009500 comp rx=0 tx=0).stop 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 shutdown_connections 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56396412f500 0x563964131920 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5639640c7330 0x5639640cd8b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5639640beb20 0x5639640cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 --2- 172.21.15.2:0/1228128396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5639640bddb0 0x5639640cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 >> 172.21.15.2:0/1228128396 conn(0x5639640854b0 msgr2=0x5639640b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.563+0000 7f2e65600640 0 client.25703 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.564+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 shutdown_connections 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.564+0000 7f2e8156c740 1 -- 172.21.15.2:0/1228128396 wait complete. 2024-03-11T18:36:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.564+0000 7f2e8156c740 20 client.25703 trim_cache size 0 max 16384 2024-03-11T18:36:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:36:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:36:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:36:26.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 msgr2=0x7f2e5c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 0x7f2e5c0a4050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2e70004af0 tx=0x7f2e70005fd0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 msgr2=0x7f2e78073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800e9f0 tx=0x7f2e6800edf0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 shutdown_connections 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f2e5c0a1b90 0x7f2e5c0a4050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e78073f00 0x7f2e781a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2e78106460 0x7f2e780739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 --2- 172.21.15.2:0/38014365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e78101390 0x7f2e78073480 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:26.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 >> 172.21.15.2:0/38014365 conn(0x7f2e780fcbd0 msgr2=0x7f2e780feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:26.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 shutdown_connections 2024-03-11T18:36:26.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:26.565+0000 7f2e7e800640 1 -- 172.21.15.2:0/38014365 wait complete. 2024-03-11T18:36:27.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 152642 9.68 MB/sec warmup 87 sec latency 61.720 ms 2024-03-11T18:36:28.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:27 smithi002 ceph-mon[31328]: pgmap v85: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:36:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:27 smithi165 ceph-mon[36634]: pgmap v85: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:36:28.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:27 smithi023 ceph-mon[36406]: pgmap v85: 129 pgs: 129 active+clean; 50 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:36:28.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 153519 9.59 MB/sec warmup 88 sec latency 50.525 ms 2024-03-11T18:36:29.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 156070 9.67 MB/sec warmup 89 sec latency 67.093 ms 2024-03-11T18:36:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:29 smithi002 ceph-mon[31328]: pgmap v86: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 400 op/s 2024-03-11T18:36:30.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:29 smithi165 ceph-mon[36634]: pgmap v86: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 400 op/s 2024-03-11T18:36:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:29 smithi023 ceph-mon[36406]: pgmap v86: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 400 op/s 2024-03-11T18:36:30.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 157082 9.59 MB/sec warmup 90 sec latency 15.624 ms 2024-03-11T18:36:31.385 INFO:tasks.workunit.client.0.smithi002.stdout: 1 159755 9.67 MB/sec warmup 91 sec latency 9.707 ms 2024-03-11T18:36:32.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:31 smithi002 ceph-mon[31328]: pgmap v87: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:31 smithi165 ceph-mon[36634]: pgmap v87: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:31 smithi023 ceph-mon[36406]: pgmap v87: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:32.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 160629 9.59 MB/sec warmup 92 sec latency 20.850 ms 2024-03-11T18:36:33.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 162913 9.63 MB/sec warmup 93 sec latency 12.860 ms 2024-03-11T18:36:34.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:33 smithi002 ceph-mon[31328]: pgmap v88: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:34.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:33 smithi165 ceph-mon[36634]: pgmap v88: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:34.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:33 smithi023 ceph-mon[36406]: pgmap v88: 129 pgs: 129 active+clean; 44 MiB data, 674 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-11T18:36:34.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 163894 9.58 MB/sec warmup 94 sec latency 58.084 ms 2024-03-11T18:36:35.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 165612 9.55 MB/sec warmup 95 sec latency 20.663 ms 2024-03-11T18:36:36.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:35 smithi002 ceph-mon[31328]: pgmap v89: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2024-03-11T18:36:36.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:35 smithi023 ceph-mon[36406]: pgmap v89: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2024-03-11T18:36:36.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:35 smithi165 ceph-mon[36634]: pgmap v89: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 391 op/s 2024-03-11T18:36:36.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 167130 9.58 MB/sec warmup 96 sec latency 10.069 ms 2024-03-11T18:36:37.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 168338 9.53 MB/sec warmup 97 sec latency 24.226 ms 2024-03-11T18:36:38.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:37 smithi002 ceph-mon[31328]: pgmap v90: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-03-11T18:36:38.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:37 smithi023 ceph-mon[36406]: pgmap v90: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-03-11T18:36:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:37 smithi165 ceph-mon[36634]: pgmap v90: 129 pgs: 129 active+clean; 42 MiB data, 665 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 1.6 MiB/s wr, 301 op/s 2024-03-11T18:36:38.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 170502 9.58 MB/sec warmup 98 sec latency 14.326 ms 2024-03-11T18:36:39.386 INFO:tasks.workunit.client.0.smithi002.stdout: 1 171196 9.49 MB/sec warmup 99 sec latency 68.692 ms 2024-03-11T18:36:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.763+0000 7fba47400640 1 -- 172.21.15.2:0/2140537200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 msgr2=0x7fba40100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.763+0000 7fba47400640 1 --2- 172.21.15.2:0/2140537200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba40100680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba34009ca0 tx=0x7fba3402c140 comp rx=0 tx=0).stop 2024-03-11T18:36:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.764+0000 7fba47400640 1 -- 172.21.15.2:0/2140537200 shutdown_connections 2024-03-11T18:36:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.764+0000 7fba47400640 1 --2- 172.21.15.2:0/2140537200 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba40105370 0x7fba4010cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.764+0000 7fba47400640 1 --2- 172.21.15.2:0/2140537200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40100c50 0x7fba40104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.764+0000 7fba47400640 1 --2- 172.21.15.2:0/2140537200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba40100680 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.764+0000 7fba47400640 1 -- 172.21.15.2:0/2140537200 >> 172.21.15.2:0/2140537200 conn(0x7fba400fbb20 msgr2=0x7fba400fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.765+0000 7fba47400640 1 -- 172.21.15.2:0/2140537200 shutdown_connections 2024-03-11T18:36:39.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.765+0000 7fba47400640 1 -- 172.21.15.2:0/2140537200 wait complete. 2024-03-11T18:36:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.767+0000 7fba47400640 1 Processor -- start 2024-03-11T18:36:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.767+0000 7fba47400640 1 -- start start 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba40105370 0x7fba4019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40199920 0x7fba40199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4019a310 con 0x7fba40105370 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4019a480 con 0x7fba40199920 2024-03-11T18:36:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba47400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4019a640 con 0x7fba401002a0 2024-03-11T18:36:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba44a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba44a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57056/0 (socket says 172.21.15.2:57056) 2024-03-11T18:36:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba44a00640 1 -- 172.21.15.2:0/3027175225 learned_addr learned my addr 172.21.15.2:0/3027175225 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.768+0000 7fba45400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40199920 0x7fba40199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.769+0000 7fba44a00640 1 -- 172.21.15.2:0/3027175225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40199920 msgr2=0x7fba40199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.769+0000 7fba44a00640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40199920 0x7fba40199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.769+0000 7fba44a00640 1 -- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba40105370 msgr2=0x7fba4019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.769+0000 7fba44a00640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba40105370 0x7fba4019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.769+0000 7fba44a00640 1 -- 172.21.15.2:0/3027175225 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34009900 con 0x7fba401002a0 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.770+0000 7fba44a00640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba34009dc0 tx=0x7fba34032f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.770+0000 7fba3d600640 1 -- 172.21.15.2:0/3027175225 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3403d3f0 con 0x7fba401002a0 2024-03-11T18:36:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.770+0000 7fba3d600640 1 -- 172.21.15.2:0/3027175225 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba3403da30 con 0x7fba401002a0 2024-03-11T18:36:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.770+0000 7fba3d600640 1 -- 172.21.15.2:0/3027175225 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3403c9c0 con 0x7fba401002a0 2024-03-11T18:36:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.772+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba401a6630 con 0x7fba401002a0 2024-03-11T18:36:39.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.772+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401a6a00 con 0x7fba401002a0 2024-03-11T18:36:39.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.778+0000 7fba3d600640 1 -- 172.21.15.2:0/3027175225 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fba3403b040 con 0x7fba401002a0 2024-03-11T18:36:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.781+0000 7fba3d600640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 0x7fba280a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.782+0000 7fba3d600640 1 -- 172.21.15.2:0/3027175225 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba340f0440 con 0x7fba401002a0 2024-03-11T18:36:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.782+0000 7fba4a225740 1 Processor -- start 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.783+0000 7fba4a225740 1 -- start start 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.783+0000 7fba3fe00640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 0x7fba280a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.783+0000 7fba4a225740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.783+0000 7fba4a225740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.783+0000 7fba4a225740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba4a225740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564908058620 con 0x5649080aeb40 2024-03-11T18:36:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba4a225740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56490800a8c0 con 0x5649080a6330 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba4a225740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564908007df0 con 0x5649080a55c0 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba44a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba44a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51266/0 (socket says 172.21.15.2:51266) 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba44a00640 1 -- 172.21.15.2:0/742983569 learned_addr learned my addr 172.21.15.2:0/742983569 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba45400640 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.784+0000 7fba3fe00640 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 -- 172.21.15.2:0/742983569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 msgr2=0x5649080a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080a5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 -- 172.21.15.2:0/742983569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 msgr2=0x5649080b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080b0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 -- 172.21.15.2:0/742983569 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564907f84b00 con 0x5649080a6330 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.785+0000 7fba44a00640 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba34005e30 tx=0x7fba3403df10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8d34cf2ed785833 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.786+0000 7fba31c00640 1 -- 172.21.15.2:0/742983569 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340c0fa0 con 0x5649080a6330 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.786+0000 7fba31c00640 1 -- 172.21.15.2:0/742983569 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba340c5560 con 0x5649080a6330 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.786+0000 7fba31c00640 1 -- 172.21.15.2:0/742983569 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340c30e0 con 0x5649080a6330 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 -- 172.21.15.2:0/742983569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 msgr2=0x5649080a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fba34005e30 tx=0x7fba3403df10 comp rx=0 tx=0).stop 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba3fe00640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 0x7fba280a41e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba400fc0f0 tx=0x7fba2c0096f0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 -- 172.21.15.2:0/742983569 shutdown_connections 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 --2- 172.21.15.2:0/742983569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080a5080 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 -- 172.21.15.2:0/742983569 >> 172.21.15.2:0/742983569 conn(0x5649080644b0 msgr2=0x5649080a02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.787+0000 7fba4a225740 1 -- 172.21.15.2:0/742983569 shutdown_connections 2024-03-11T18:36:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.788+0000 7fba4a225740 1 -- 172.21.15.2:0/742983569 wait complete. 2024-03-11T18:36:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.788+0000 7fba4a225740 1 -- start start 2024-03-11T18:36:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 Processor -- start 2024-03-11T18:36:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080abb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 0x5649080ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649080035d0 con 0x5649080a6330 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564907fa8b50 con 0x5649080aeb40 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.789+0000 7fba4a225740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564907fa87f0 con 0x5649080a55c0 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.790+0000 7fba4a225740 10 client.0 _ll_register_callbacks cb 0x564907fbc550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.790+0000 7fba45400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.790+0000 7fba45400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51272/0 (socket says 172.21.15.2:51272) 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.790+0000 7fba45400640 1 -- 172.21.15.2:0/562284429 learned_addr learned my addr 172.21.15.2:0/562284429 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.790+0000 7fba45400640 1 -- 172.21.15.2:0/562284429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 msgr2=0x5649080abb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.791+0000 7fba44a00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.791+0000 7fba3fe00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 0x5649080ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.791+0000 7fba45400640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.791+0000 7fba45400640 1 -- 172.21.15.2:0/562284429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 msgr2=0x5649080ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.791+0000 7fba45400640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 0x5649080ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.792+0000 7fba45400640 1 -- 172.21.15.2:0/562284429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401a6a00 con 0x5649080aeb40 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.792+0000 7fba3fe00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 0x5649080ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba45400640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fba38010420 tx=0x7fba380108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cde15aa7d48b903c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba26000640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38012920 con 0x5649080aeb40 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba38016070 con 0x5649080aeb40 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba44a00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.793+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564907fd5cd0 con 0x5649080aeb40 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba38012a80 con 0x5649080aeb40 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 10 client.25712 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:39.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564908001c50 con 0x5649080aeb40 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564907fd5cd0 con 0x5649080aeb40 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fba38011440 con 0x5649080aeb40 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 10 client.25712 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 20 client.25712 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649080cf920 0x5649080d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 4 client.25712 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.794+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564907ff9ce0 con 0x5649080cf920 2024-03-11T18:36:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.795+0000 7fba44a00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649080cf920 0x5649080d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.795+0000 7fba44a00640 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649080cf920 0x5649080d1d40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=386346dd99f691aa server_cookie=34b2ba3ab0f6bf69 in_seq=0 out_seq=0 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.795+0000 7fba26000640 10 client.25712 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.796+0000 7fba26000640 1 -- 172.21.15.2:0/562284429 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564907ff9ce0 con 0x5649080cf920 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.796+0000 7fba26000640 10 client.25712 handle_command_reply: tid=0 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.796+0000 7fba4a225740 1 client.25712 shutdown 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 msgr2=0x5649080ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fba38010420 tx=0x7fba380108e0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 shutdown_connections 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649080cf920 0x5649080d1d40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080aeb40 0x5649080ac600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a6330 0x5649080ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 --2- 172.21.15.2:0/562284429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a55c0 0x5649080abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba26000640 0 client.25712 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 >> 172.21.15.2:0/562284429 conn(0x5649080644b0 msgr2=0x5649080a1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 shutdown_connections 2024-03-11T18:36:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 1 -- 172.21.15.2:0/562284429 wait complete. 2024-03-11T18:36:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.797+0000 7fba4a225740 20 client.25712 trim_cache size 0 max 16384 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 Processor -- start 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 -- start start 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a6330 0x5649080c6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564907d4be60 con 0x5649080aeb40 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564907fd5d00 con 0x5649080a55c0 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.813+0000 7fba4a225740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649080cbc90 con 0x5649080a6330 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51284/0 (socket says 172.21.15.2:51284) 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 -- 172.21.15.2:0/1349875250 learned_addr learned my addr 172.21.15.2:0/1349875250 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba3fe00640 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a6330 0x5649080c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba45400640 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 -- 172.21.15.2:0/1349875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a6330 msgr2=0x5649080c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a6330 0x5649080c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 -- 172.21.15.2:0/1349875250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 msgr2=0x5649080c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 -- 172.21.15.2:0/1349875250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564907f9e530 con 0x5649080a55c0 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.814+0000 7fba44a00640 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fba3404b000 tx=0x7fba340eda50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3789357f443544f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.815+0000 7fba31200640 1 -- 172.21.15.2:0/1349875250 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340c0fa0 con 0x5649080a55c0 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.815+0000 7fba31200640 1 -- 172.21.15.2:0/1349875250 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba340bce50 con 0x5649080a55c0 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.815+0000 7fba31200640 1 -- 172.21.15.2:0/1349875250 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340c30b0 con 0x5649080a55c0 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- 172.21.15.2:0/1349875250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 msgr2=0x5649080ab460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fba3404b000 tx=0x7fba340eda50 comp rx=0 tx=0).stop 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- 172.21.15.2:0/1349875250 shutdown_connections 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080aeb40 0x5649080c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080a6330 0x5649080c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 --2- 172.21.15.2:0/1349875250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a55c0 0x5649080ab460 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- 172.21.15.2:0/1349875250 >> 172.21.15.2:0/1349875250 conn(0x5649080644b0 msgr2=0x5649080a1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- 172.21.15.2:0/1349875250 shutdown_connections 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- 172.21.15.2:0/1349875250 wait complete. 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 -- start start 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.816+0000 7fba4a225740 1 Processor -- start 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a55c0 0x5649080c6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 0x5649080c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649080c52b0 con 0x5649080a55c0 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649080c5420 con 0x5649080a6330 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649080c55e0 con 0x5649080aeb40 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba4a225740 10 client.0 _ll_register_callbacks cb 0x7fba38005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:51296/0 (socket says 172.21.15.2:51296) 2024-03-11T18:36:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 -- 172.21.15.2:0/3480805815 learned_addr learned my addr 172.21.15.2:0/3480805815 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba45400640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 0x5649080c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba44a00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a55c0 0x5649080c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 -- 172.21.15.2:0/3480805815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 msgr2=0x5649080c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 0x5649080c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 -- 172.21.15.2:0/3480805815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a55c0 msgr2=0x5649080c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a55c0 0x5649080c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.817+0000 7fba3fe00640 1 -- 172.21.15.2:0/3480805815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564907f84b00 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba45400640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 0x5649080c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba3fe00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba4019b110 tx=0x7fba2c0047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa6a2a31b64b9b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba26000640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00c070 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba2c011040 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.818+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564908028d30 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba2c016e90 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 10 client.25718 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564907fc42c0 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564908028d30 con 0x5649080a6330 2024-03-11T18:36:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fba2c022430 con 0x5649080a6330 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 10 client.25718 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 20 client.25718 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649081272d0 0x5649081296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 4 client.25718 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564907d03600 con 0x5649081272d0 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.819+0000 7fba44a00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649081272d0 0x5649081296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.820+0000 7fba44a00640 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649081272d0 0x5649081296f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30452584d6309c42 server_cookie=4955d7369ff26b13 in_seq=0 out_seq=0 2024-03-11T18:36:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.820+0000 7fba26000640 10 client.25718 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:39.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.989+0000 7fba26000640 1 -- 172.21.15.2:0/3480805815 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564907d03600 con 0x5649081272d0 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.989+0000 7fba26000640 10 client.25718 handle_command_reply: tid=0 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.989+0000 7fba4a225740 1 client.25718 shutdown 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.989+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 msgr2=0x5649080c6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.989+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba4019b110 tx=0x7fba2c0047e0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 shutdown_connections 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5649081272d0 0x5649081296f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5649080aeb40 0x5649080c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5649080a6330 0x5649080c6b80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 --2- 172.21.15.2:0/3480805815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5649080a55c0 0x5649080c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba26000640 0 client.25718 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 >> 172.21.15.2:0/3480805815 conn(0x5649080644b0 msgr2=0x5649080a13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 shutdown_connections 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 1 -- 172.21.15.2:0/3480805815 wait complete. 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.990+0000 7fba4a225740 20 client.25718 trim_cache size 0 max 16384 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:36:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 msgr2=0x7fba280a41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 0x7fba280a41e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba400fc0f0 tx=0x7fba2c0096f0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 msgr2=0x7fba4019f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fba34009dc0 tx=0x7fba34032f30 comp rx=0 tx=0).stop 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 shutdown_connections 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba40199920 0x7fba40199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba280a1d20 0x7fba280a41e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba40105370 0x7fba4019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 --2- 172.21.15.2:0/3027175225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba401002a0 0x7fba4019f770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:39.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 >> 172.21.15.2:0/3027175225 conn(0x7fba400fbb20 msgr2=0x7fba400fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:39.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 shutdown_connections 2024-03-11T18:36:39.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:39.991+0000 7fba47400640 1 -- 172.21.15.2:0/3027175225 wait complete. 2024-03-11T18:36:40.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:39 smithi002 ceph-mon[31328]: pgmap v91: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-11T18:36:40.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:39 smithi165 ceph-mon[36634]: pgmap v91: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-11T18:36:40.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:39 smithi023 ceph-mon[36406]: pgmap v91: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-11T18:36:40.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 173449 9.52 MB/sec warmup 100 sec latency 67.567 ms 2024-03-11T18:36:41.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 174709 9.49 MB/sec warmup 101 sec latency 16.032 ms 2024-03-11T18:36:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:41 smithi002 ceph-mon[31328]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:41 smithi165 ceph-mon[36634]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:41 smithi023 ceph-mon[36406]: pgmap v92: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:42.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 177147 9.53 MB/sec warmup 102 sec latency 13.730 ms 2024-03-11T18:36:43.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 178252 9.49 MB/sec warmup 103 sec latency 13.717 ms 2024-03-11T18:36:43.957 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:43 smithi165 ceph-mon[36634]: pgmap v93: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:44.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:43 smithi002 ceph-mon[31328]: pgmap v93: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:44.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:43 smithi023 ceph-mon[36406]: pgmap v93: 129 pgs: 129 active+clean; 46 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-03-11T18:36:44.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 180386 9.52 MB/sec warmup 104 sec latency 62.069 ms 2024-03-11T18:36:45.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 181718 9.49 MB/sec warmup 105 sec latency 15.224 ms 2024-03-11T18:36:46.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:45 smithi002 ceph-mon[31328]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 355 op/s 2024-03-11T18:36:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:45 smithi165 ceph-mon[36634]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 355 op/s 2024-03-11T18:36:46.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:45 smithi023 ceph-mon[36406]: pgmap v94: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 355 op/s 2024-03-11T18:36:46.387 INFO:tasks.workunit.client.0.smithi002.stdout: 1 184078 9.52 MB/sec warmup 106 sec latency 13.956 ms 2024-03-11T18:36:47.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 185236 9.49 MB/sec warmup 107 sec latency 22.713 ms 2024-03-11T18:36:48.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:47 smithi002 ceph-mon[31328]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.1 MiB/s wr, 271 op/s 2024-03-11T18:36:48.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:47 smithi165 ceph-mon[36634]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.1 MiB/s wr, 271 op/s 2024-03-11T18:36:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:47 smithi023 ceph-mon[36406]: pgmap v95: 129 pgs: 129 active+clean; 41 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.1 MiB/s wr, 271 op/s 2024-03-11T18:36:48.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 187051 9.48 MB/sec warmup 108 sec latency 16.125 ms 2024-03-11T18:36:49.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 188419 9.49 MB/sec warmup 109 sec latency 66.429 ms 2024-03-11T18:36:50.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:49 smithi002 ceph-mon[31328]: pgmap v96: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:36:50.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:49 smithi023 ceph-mon[36406]: pgmap v96: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:36:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:49 smithi165 ceph-mon[36634]: pgmap v96: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-11T18:36:50.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 189985 9.45 MB/sec warmup 110 sec latency 37.109 ms 2024-03-11T18:36:51.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 191819 9.49 MB/sec warmup 111 sec latency 22.008 ms 2024-03-11T18:36:52.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:51 smithi002 ceph-mon[31328]: pgmap v97: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:52.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:51 smithi023 ceph-mon[36406]: pgmap v97: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:52.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:51 smithi165 ceph-mon[36634]: pgmap v97: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:52.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 192733 9.42 MB/sec warmup 112 sec latency 45.015 ms 2024-03-11T18:36:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 -- 172.21.15.2:0/1420604576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c58107f60 msgr2=0x7f3c58114420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 --2- 172.21.15.2:0/1420604576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c58107f60 0x7f3c58114420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009cd0 tx=0x7f3c5002c0f0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 -- 172.21.15.2:0/1420604576 shutdown_connections 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 --2- 172.21.15.2:0/1420604576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c58107f60 0x7f3c58114420 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 --2- 172.21.15.2:0/1420604576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c581075a0 0x7f3c58107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 --2- 172.21.15.2:0/1420604576 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c5810d660 0x7f3c5810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 -- 172.21.15.2:0/1420604576 >> 172.21.15.2:0/1420604576 conn(0x7f3c58076d00 msgr2=0x7f3c58079120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 -- 172.21.15.2:0/1420604576 shutdown_connections 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.141+0000 7f3c57200640 1 -- 172.21.15.2:0/1420604576 wait complete. 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.143+0000 7f3c57200640 1 Processor -- start 2024-03-11T18:36:53.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.143+0000 7f3c57200640 1 -- start start 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 0x7f3c581a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c5810d660 0x7f3c5819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5819dff0 con 0x7f3c58107f60 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5819e160 con 0x7f3c581075a0 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c57200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5819e320 con 0x7f3c5810d660 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58158/0 (socket says 172.21.15.2:58158) 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 -- 172.21.15.2:0/200580566 learned_addr learned my addr 172.21.15.2:0/200580566 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:53.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c56800640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c5810d660 0x7f3c5819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55400640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 0x7f3c581a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 -- 172.21.15.2:0/200580566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c5810d660 msgr2=0x7f3c5819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c5810d660 0x7f3c5819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 -- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 msgr2=0x7f3c581a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 0x7f3c581a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55e00640 1 -- 172.21.15.2:0/200580566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c50009900 con 0x7f3c581075a0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.144+0000 7f3c55400640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 0x7f3c581a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c55e00640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800e9f0 tx=0x7f3c4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c4ea00640 1 -- 172.21.15.2:0/200580566 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48009700 con 0x7f3c581075a0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c4ea00640 1 -- 172.21.15.2:0/200580566 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c4800cd80 con 0x7f3c581075a0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c4ea00640 1 -- 172.21.15.2:0/200580566 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c48018920 con 0x7f3c581075a0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5819e600 con 0x7f3c581075a0 2024-03-11T18:36:53.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.145+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c581aa890 con 0x7f3c581075a0 2024-03-11T18:36:53.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.149+0000 7f3c5f9c3740 1 Processor -- start 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 -- start start 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c4ea00640 1 -- 172.21.15.2:0/200580566 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f3c48018a80 con 0x7f3c581075a0 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311915d60 0x556311914ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55631191e570 0x556311920960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563118d0620 con 0x55631191e570 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563118828c0 con 0x556311914ff0 2024-03-11T18:36:53.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55631187fdf0 con 0x556311915d60 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.150+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c56800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311915d60 0x556311914ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58168/0 (socket says 172.21.15.2:58168) 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 -- 172.21.15.2:0/3677354845 learned_addr learned my addr 172.21.15.2:0/3677354845 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55400640 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55631191e570 0x556311920960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 -- 172.21.15.2:0/3677354845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311915d60 msgr2=0x556311914ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311915d60 0x556311914ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 -- 172.21.15.2:0/3677354845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55631191e570 msgr2=0x556311920960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55631191e570 0x556311920960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 -- 172.21.15.2:0/3677354845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563117fcb00 con 0x556311914ff0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.151+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cb1a0 tx=0x7f3c480d67d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d08aab4f3ddaf68 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.152+0000 7f3c43000640 1 -- 172.21.15.2:0/3677354845 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c480d4680 con 0x556311914ff0 2024-03-11T18:36:53.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.152+0000 7f3c43000640 1 -- 172.21.15.2:0/3677354845 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c480d4cc0 con 0x556311914ff0 2024-03-11T18:36:53.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.152+0000 7f3c43000640 1 -- 172.21.15.2:0/3677354845 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c480dedc0 con 0x556311914ff0 2024-03-11T18:36:53.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.152+0000 7f3c4ea00640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 0x7f3c3c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.152+0000 7f3c4ea00640 1 -- 172.21.15.2:0/200580566 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3c48014070 con 0x7f3c581075a0 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c56800640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 0x7f3c3c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3677354845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 msgr2=0x556311915470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c480cb1a0 tx=0x7f3c480d67d0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3677354845 shutdown_connections 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55631191e570 0x556311920960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311915470 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3677354845 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311915d60 0x556311914ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3677354845 >> 172.21.15.2:0/3677354845 conn(0x5563118dc4b0 msgr2=0x55631190fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3677354845 shutdown_connections 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3677354845 wait complete. 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.153+0000 7f3c56800640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 0x7f3c3c0a40c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c5002c630 tx=0x7f3c50005d80 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.154+0000 7f3c5f9c3740 1 -- start start 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.154+0000 7f3c5f9c3740 1 Processor -- start 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x55631191bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55631191e570 0x55631191c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55631187b5d0 con 0x556311915d60 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556311820b50 con 0x55631191e570 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563118207f0 con 0x556311914ff0 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c5f9c3740 10 client.0 _ll_register_callbacks cb 0x556311834550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:53.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c56800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55631191e570 0x55631191c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42462/0 (socket says 172.21.15.2:42462) 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 -- 172.21.15.2:0/461441649 learned_addr learned my addr 172.21.15.2:0/461441649 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55400640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x55631191bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 -- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55631191e570 msgr2=0x55631191c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55631191e570 0x55631191c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 -- 172.21.15.2:0/461441649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 msgr2=0x55631191bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x55631191bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.155+0000 7f3c55e00640 1 -- 172.21.15.2:0/461441649 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c581aa890 con 0x556311914ff0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c55e00640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c48096ba0 tx=0x7f3c48010e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be3dfca7b637d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c37400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c480de6e0 con 0x556311914ff0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3c480de3e0 con 0x556311914ff0 2024-03-11T18:36:53.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55631184dcd0 con 0x556311914ff0 2024-03-11T18:36:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c480969e0 con 0x556311914ff0 2024-03-11T18:36:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c5f9c3740 10 client.25726 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556311879c50 con 0x556311914ff0 2024-03-11T18:36:53.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.156+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c480b2070 con 0x556311914ff0 2024-03-11T18:36:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3c48096040 con 0x556311914ff0 2024-03-11T18:36:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 10 client.25726 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 10 client.25726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 20 client.25726 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:53.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55631192e110 0x556311930500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 4 client.25726 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556311871ce0 con 0x55631192e110 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.157+0000 7f3c55400640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55631192e110 0x556311930500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.158+0000 7f3c55400640 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55631192e110 0x556311930500 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8d83eb776bfe3d server_cookie=f553447f8622d922 in_seq=0 out_seq=0 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.158+0000 7f3c37400640 10 client.25726 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c37400640 1 -- 172.21.15.2:0/461441649 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556311871ce0 con 0x55631192e110 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c37400640 10 client.25726 handle_command_reply: tid=0 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 client.25726 shutdown 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 msgr2=0x55631191b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3c48096ba0 tx=0x7f3c48010e50 comp rx=0 tx=0).stop 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 shutdown_connections 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55631192e110 0x556311930500 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55631191e570 0x55631191c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x55631191bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c37400640 0 client.25726 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/461441649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556311914ff0 0x55631191b5e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 >> 172.21.15.2:0/461441649 conn(0x5563118dc4b0 msgr2=0x556311911a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 shutdown_connections 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.159+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/461441649 wait complete. 2024-03-11T18:36:53.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.160+0000 7f3c5f9c3740 20 client.25726 trim_cache size 0 max 16384 2024-03-11T18:36:53.178 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 Processor -- start 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 -- start start 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311914ff0 0x55631191dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x556311918840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563115c3e60 con 0x556311914ff0 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556311914e70 con 0x556311915d60 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556311914620 con 0x55631191e570 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.175+0000 7f3c55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58190/0 (socket says 172.21.15.2:58190) 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 -- 172.21.15.2:0/94453302 learned_addr learned my addr 172.21.15.2:0/94453302 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c56800640 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x556311918840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55e00640 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311914ff0 0x55631191dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 -- 172.21.15.2:0/94453302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 msgr2=0x556311918840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x556311918840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 -- 172.21.15.2:0/94453302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311914ff0 msgr2=0x55631191dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311914ff0 0x55631191dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 -- 172.21.15.2:0/94453302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556311816530 con 0x556311915d60 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c55400640 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009c80 tx=0x7f3c44009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ad8f9074a272fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.176+0000 7f3c42600640 1 -- 172.21.15.2:0/94453302 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44018040 con 0x556311915d60 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c42600640 1 -- 172.21.15.2:0/94453302 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c44021020 con 0x556311915d60 2024-03-11T18:36:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c42600640 1 -- 172.21.15.2:0/94453302 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44012d80 con 0x556311915d60 2024-03-11T18:36:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/94453302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 msgr2=0x55631191be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009c80 tx=0x7f3c44009f90 comp rx=0 tx=0).stop 2024-03-11T18:36:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/94453302 shutdown_connections 2024-03-11T18:36:53.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x556311918840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311915d60 0x55631191be80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/94453302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311914ff0 0x55631191dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/94453302 >> 172.21.15.2:0/94453302 conn(0x5563118dc4b0 msgr2=0x556311919eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/94453302 shutdown_connections 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.177+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/94453302 wait complete. 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.178+0000 7f3c5f9c3740 1 -- start start 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.178+0000 7f3c5f9c3740 1 Processor -- start 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.178+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.178+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x556311924370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c5f9c3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x5563119248b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563119845f0 con 0x556311915d60 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556311984730 con 0x556311914ff0 2024-03-11T18:36:53.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c5f9c3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556311984870 con 0x55631191e570 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c5f9c3740 10 client.0 _ll_register_callbacks cb 0x7f3c440095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x556311924370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58194/0 (socket says 172.21.15.2:58194) 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 -- 172.21.15.2:0/3380985154 learned_addr learned my addr 172.21.15.2:0/3380985154 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c56800640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x5563119248b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 -- 172.21.15.2:0/3380985154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 msgr2=0x5563119248b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x5563119248b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 -- 172.21.15.2:0/3380985154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 msgr2=0x556311924370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x556311924370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.179+0000 7f3c55e00640 1 -- 172.21.15.2:0/3380985154 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c440099b0 con 0x556311914ff0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c55e00640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3c4809a440 tx=0x7f3c4800cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe64ce249ea45cc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:36:53.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c480d6d80 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3c480a2ee0 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563118a0d30 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c480d12f0 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c5f9c3740 10 client.25736 fetch_fsmap learned FSMap version 26 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55631183c2c0 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c480b8020 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f3c48096040 con 0x556311914ff0 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c5f9c3740 10 client.25736 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.180+0000 7f3c5f9c3740 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c5f9c3740 20 client.25736 populate_metadata read hostname 'smithi002' 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556311985b40 0x556311987f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c5f9c3740 4 client.25736 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:36:53.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55631157b600 con 0x556311985b40 2024-03-11T18:36:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c55400640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556311985b40 0x556311987f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:36:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c55400640 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556311985b40 0x556311987f30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79a444c538aeb69c server_cookie=3a0fec746e9ab608 in_seq=0 out_seq=0 2024-03-11T18:36:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.181+0000 7f3c37400640 10 client.25736 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c37400640 1 -- 172.21.15.2:0/3380985154 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55631157b600 con 0x556311985b40 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c37400640 10 client.25736 handle_command_reply: tid=0 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 client.25736 shutdown 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 msgr2=0x556311923e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3c4809a440 tx=0x7f3c4800cef0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 shutdown_connections 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556311985b40 0x556311987f30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.369+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55631191e570 0x5563119248b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556311915d60 0x556311924370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 1 --2- 172.21.15.2:0/3380985154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556311914ff0 0x556311923e30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 >> 172.21.15.2:0/3380985154 conn(0x5563118dc4b0 msgr2=0x55631190e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c37400640 0 client.25736 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 shutdown_connections 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 1 -- 172.21.15.2:0/3380985154 wait complete. 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.370+0000 7f3c5f9c3740 20 client.25736 trim_cache size 0 max 16384 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:36:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 msgr2=0x7f3c3c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 0x7f3c3c0a40c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c5002c630 tx=0x7f3c50005d80 comp rx=0 tx=0).stop 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 msgr2=0x7f3c581a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800e9f0 tx=0x7f3c4800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 shutdown_connections 2024-03-11T18:36:53.374 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3c5810d660 0x7f3c5819d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3c3c0a1c00 0x7f3c3c0a40c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3c58107f60 0x7f3c581a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 --2- 172.21.15.2:0/200580566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c581075a0 0x7f3c581a37f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 >> 172.21.15.2:0/200580566 conn(0x7f3c58076d00 msgr2=0x7f3c58078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 shutdown_connections 2024-03-11T18:36:53.375 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:36:53.371+0000 7f3c57200640 1 -- 172.21.15.2:0/200580566 wait complete. 2024-03-11T18:36:53.388 INFO:tasks.workunit.client.0.smithi002.stdout: 1 194968 9.46 MB/sec warmup 113 sec latency 101.838 ms 2024-03-11T18:36:54.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:53 smithi002 ceph-mon[31328]: pgmap v98: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:54.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:53 smithi165 ceph-mon[36634]: pgmap v98: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:54.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:53 smithi023 ceph-mon[36406]: pgmap v98: 129 pgs: 129 active+clean; 44 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2024-03-11T18:36:54.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 195902 9.41 MB/sec warmup 114 sec latency 35.161 ms 2024-03-11T18:36:55.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 198263 9.44 MB/sec warmup 115 sec latency 21.612 ms 2024-03-11T18:36:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:55 smithi002 ceph-mon[31328]: pgmap v99: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 473 op/s 2024-03-11T18:36:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:55 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:36:56.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:55 smithi165 ceph-mon[36634]: pgmap v99: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 473 op/s 2024-03-11T18:36:56.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:55 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:36:56.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:55 smithi023 ceph-mon[36406]: pgmap v99: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 473 op/s 2024-03-11T18:36:56.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:55 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:36:56.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 199439 9.42 MB/sec warmup 116 sec latency 15.887 ms 2024-03-11T18:36:57.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:56 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:36:57.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:56 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:36:57.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:56 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:36:57.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:56 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:36:57.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:56 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:36:57.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:56 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:36:57.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:56 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:36:57.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:56 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:36:57.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:56 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:36:57.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 201405 9.43 MB/sec warmup 117 sec latency 19.010 ms 2024-03-11T18:36:58.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 202738 9.41 MB/sec warmup 118 sec latency 13.222 ms 2024-03-11T18:36:58.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:36:58 smithi002 ceph-mon[31328]: pgmap v100: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:36:58.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:36:58 smithi165 ceph-mon[36634]: pgmap v100: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:36:58.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:36:58 smithi023 ceph-mon[36406]: pgmap v100: 129 pgs: 129 active+clean; 43 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:36:59.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 204047 9.38 MB/sec warmup 119 sec latency 33.706 ms 2024-03-11T18:37:00.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:00 smithi002 ceph-mon[31328]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-03-11T18:37:00.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:00 smithi165 ceph-mon[36634]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-03-11T18:37:00.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:00 smithi023 ceph-mon[36406]: pgmap v101: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-03-11T18:37:01.389 INFO:tasks.workunit.client.0.smithi002.stdout: 1 206744 1.42 MB/sec execute 1 sec latency 72.636 ms 2024-03-11T18:37:02.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 206961 1.22 MB/sec execute 2 sec latency 101.914 ms 2024-03-11T18:37:02.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:02 smithi002 ceph-mon[31328]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:02.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:02 smithi165 ceph-mon[36634]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:02.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:02 smithi023 ceph-mon[36406]: pgmap v102: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:03.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 209077 4.91 MB/sec execute 3 sec latency 15.235 ms 2024-03-11T18:37:04.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 210086 4.99 MB/sec execute 4 sec latency 79.730 ms 2024-03-11T18:37:04.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:04 smithi002 ceph-mon[31328]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:04 smithi165 ceph-mon[36634]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:04.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:04 smithi023 ceph-mon[36406]: pgmap v103: 129 pgs: 129 active+clean; 46 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 288 op/s 2024-03-11T18:37:05.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 212192 6.39 MB/sec execute 5 sec latency 43.315 ms 2024-03-11T18:37:06.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 213504 6.46 MB/sec execute 6 sec latency 36.393 ms 2024-03-11T18:37:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:06 smithi002 ceph-mon[31328]: pgmap v104: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 -- 172.21.15.2:0/3961290501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84111f70 msgr2=0x7f4b84114360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 --2- 172.21.15.2:0/3961290501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84111f70 0x7f4b84114360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009900 tx=0x7f4b7c02bcc0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 -- 172.21.15.2:0/3961290501 shutdown_connections 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 --2- 172.21.15.2:0/3961290501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84111f70 0x7f4b84114360 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 --2- 172.21.15.2:0/3961290501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073e40 0x7f4b84111a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 --2- 172.21.15.2:0/3961290501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84073520 0x7f4b84073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 -- 172.21.15.2:0/3961290501 >> 172.21.15.2:0/3961290501 conn(0x7f4b840fca30 msgr2=0x7f4b840fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 -- 172.21.15.2:0/3961290501 shutdown_connections 2024-03-11T18:37:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.533+0000 7f4b83200640 1 -- 172.21.15.2:0/3961290501 wait complete. 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.534+0000 7f4b83200640 1 Processor -- start 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.534+0000 7f4b83200640 1 -- start start 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073520 0x7f4b841a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84111f70 0x7f4b8419d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8419e020 con 0x7f4b84111f70 2024-03-11T18:37:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8419e190 con 0x7f4b84073e40 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b83200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8419e350 con 0x7f4b84073520 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46188/0 (socket says 172.21.15.2:46188) 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 -- 172.21.15.2:0/1584316979 learned_addr learned my addr 172.21.15.2:0/1584316979 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b82800640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84111f70 0x7f4b8419d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b81e00640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073520 0x7f4b841a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 -- 172.21.15.2:0/1584316979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073520 msgr2=0x7f4b841a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073520 0x7f4b841a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 -- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84111f70 msgr2=0x7f4b8419d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84111f70 0x7f4b8419d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.535+0000 7f4b79400640 1 -- 172.21.15.2:0/1584316979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c009560 con 0x7f4b84073e40 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b79400640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9f0 tx=0x7f4b6c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b7b400640 1 -- 172.21.15.2:0/1584316979 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c009720 con 0x7f4b84073e40 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b7b400640 1 -- 172.21.15.2:0/1584316979 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b6c00cd80 con 0x7f4b84073e40 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b7b400640 1 -- 172.21.15.2:0/1584316979 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c010690 con 0x7f4b84073e40 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8419e630 con 0x7f4b84073e40 2024-03-11T18:37:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.536+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b841aa8c0 con 0x7f4b84073e40 2024-03-11T18:37:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.539+0000 7f4b7b400640 1 -- 172.21.15.2:0/1584316979 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f4b6c0046c0 con 0x7f4b84073e40 2024-03-11T18:37:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 Processor -- start 2024-03-11T18:37:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 -- start start 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00fd570 0x5622c00ff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c00af620 con 0x5622c00fd570 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c00618c0 con 0x5622c00f4d60 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.540+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c005edf0 con 0x5622c00f3ff0 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46196/0 (socket says 172.21.15.2:46196) 2024-03-11T18:37:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 -- 172.21.15.2:0/2523318909 learned_addr learned my addr 172.21.15.2:0/2523318909 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b81e00640 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b79400640 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00fd570 0x5622c00ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 -- 172.21.15.2:0/2523318909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 msgr2=0x5622c00f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 -- 172.21.15.2:0/2523318909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00fd570 msgr2=0x5622c00ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00fd570 0x5622c00ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 -- 172.21.15.2:0/2523318909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622bffdbb00 con 0x5622c00f4d60 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b7b400640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 0x7f4b680a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.541+0000 7f4b82800640 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c005ab0 tx=0x7f4b7c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a2f90c7090cfc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.542+0000 7f4b7b400640 1 -- 172.21.15.2:0/1584316979 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f4b6c014070 con 0x7f4b84073e40 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.542+0000 7f4b81e00640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 0x7f4b680a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.542+0000 7f4b73400640 1 -- 172.21.15.2:0/2523318909 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c004800 con 0x5622c00f4d60 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.542+0000 7f4b73400640 1 -- 172.21.15.2:0/2523318909 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b7c004960 con 0x5622c00f4d60 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.542+0000 7f4b73400640 1 -- 172.21.15.2:0/2523318909 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c005050 con 0x5622c00f4d60 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2523318909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 msgr2=0x5622c00f3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.547 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b81e00640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 0x7f4b680a40c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x5622c00f54c0 tx=0x7f4b74009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c005ab0 tx=0x7f4b7c02c660 comp rx=0 tx=0).stop 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2523318909 shutdown_connections 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00fd570 0x5622c00ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2523318909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f3ab0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2523318909 >> 172.21.15.2:0/2523318909 conn(0x5622c00bb4b0 msgr2=0x5622c00eece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2523318909 shutdown_connections 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.543+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2523318909 wait complete. 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.544+0000 7f4b8bf8c740 1 -- start start 2024-03-11T18:37:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.544+0000 7f4b8bf8c740 1 Processor -- start 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f3ff0 0x5622c00fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00fd570 0x5622c00fb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c005a5d0 con 0x5622c00f3ff0 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bffffb50 con 0x5622c00fd570 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bffff7f0 con 0x5622c00f4d60 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b8bf8c740 10 client.0 _ll_register_callbacks cb 0x5622c0013550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f3ff0 0x5622c00fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:55794/0 (socket says 172.21.15.2:55794) 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 -- 172.21.15.2:0/1555968047 learned_addr learned my addr 172.21.15.2:0/1555968047 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b82800640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00fd570 0x5622c00fb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 -- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00fd570 msgr2=0x5622c00fb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00fd570 0x5622c00fb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 -- 172.21.15.2:0/1555968047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f3ff0 msgr2=0x5622c00fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f3ff0 0x5622c00fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.545+0000 7f4b79400640 1 -- 172.21.15.2:0/1555968047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b84114c90 con 0x5622c00f4d60 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b79400640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9c0 tx=0x7f4b6c09a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=295b83a2c8d0e231 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b67e00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0cb640 con 0x5622c00f4d60 2024-03-11T18:37:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f4b6c0d1020 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622c002ccd0 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b6c09ad90 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b8bf8c740 10 client.25744 fetch_fsmap learned FSMap version 26 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.546+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5622c0058c50 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5622c002ccd0 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4b6c0d3070 con 0x5622c00f4d60 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 10 client.25744 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 10 client.25744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 20 client.25744 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c010d540 0x5622c010f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 4 client.25744 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622c0050ce0 con 0x5622c010d540 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.547+0000 7f4b81e00640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c010d540 0x5622c010f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.548+0000 7f4b81e00640 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c010d540 0x5622c010f960 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aeffa19d65a5945 server_cookie=6e3c241aa3ddb87a in_seq=0 out_seq=0 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.548+0000 7f4b67e00640 10 client.25744 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b67e00640 1 -- 172.21.15.2:0/1555968047 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5622c0050ce0 con 0x5622c010d540 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b67e00640 10 client.25744 handle_command_reply: tid=0 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 client.25744 shutdown 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 msgr2=0x5622c00fab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9c0 tx=0x7f4b6c09a460 comp rx=0 tx=0).stop 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 shutdown_connections 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c010d540 0x5622c010f960 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00fd570 0x5622c00fb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f4d60 0x5622c00fab40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/1555968047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f3ff0 0x5622c00fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 >> 172.21.15.2:0/1555968047 conn(0x5622c00bb4b0 msgr2=0x5622c00f0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b67e00640 0 client.25744 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 shutdown_connections 2024-03-11T18:37:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/1555968047 wait complete. 2024-03-11T18:37:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.549+0000 7f4b8bf8c740 20 client.25744 trim_cache size 0 max 16384 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.569+0000 7f4b8bf8c740 1 Processor -- start 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.569+0000 7f4b8bf8c740 1 -- start start 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.569+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.569+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f4d60 0x5622c0104870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622bfda2e60 con 0x5622c00f4d60 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0050ce0 con 0x5622c00bb4b0 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c002ccd0 con 0x5622c00f3ff0 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b81e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b81e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46226/0 (socket says 172.21.15.2:46226) 2024-03-11T18:37:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:55806/0 (socket says 172.21.15.2:55806) 2024-03-11T18:37:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b81e00640 1 -- 172.21.15.2:0/3700127823 learned_addr learned my addr 172.21.15.2:0/3700127823 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:06.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b82800640 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f4d60 0x5622c0104870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 -- 172.21.15.2:0/3700127823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 msgr2=0x5622c00fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 -- 172.21.15.2:0/3700127823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f4d60 msgr2=0x5622c0104870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f4d60 0x5622c0104870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b79400640 1 -- 172.21.15.2:0/3700127823 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622bfff5530 con 0x5622c00f3ff0 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.570+0000 7f4b81e00640 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.571+0000 7f4b79400640 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c097060 tx=0x7f4b6c09a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7b4e7ca8d177b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.571+0000 7f4b72a00640 1 -- 172.21.15.2:0/3700127823 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0d1240 con 0x5622c00f3ff0 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.571+0000 7f4b72a00640 1 -- 172.21.15.2:0/3700127823 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b6c0b30d0 con 0x5622c00f3ff0 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.571+0000 7f4b72a00640 1 -- 172.21.15.2:0/3700127823 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0a2c00 con 0x5622c00f3ff0 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/3700127823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 msgr2=0x5622c0104330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c097060 tx=0x7f4b6c09a590 comp rx=0 tx=0).stop 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/3700127823 shutdown_connections 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00f4d60 0x5622c0104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c0104330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/3700127823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00bb4b0 0x5622c00fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/3700127823 >> 172.21.15.2:0/3700127823 conn(0x5622c00ffb40 msgr2=0x5622c00ee810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/3700127823 shutdown_connections 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.572+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/3700127823 wait complete. 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.573+0000 7f4b8bf8c740 1 -- start start 2024-03-11T18:37:06.577 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.573+0000 7f4b8bf8c740 1 Processor -- start 2024-03-11T18:37:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.573+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00bb4b0 0x5622c00f7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0163a90 con 0x5622c00bb4b0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0163c00 con 0x5622c00f4d60 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c0163dc0 con 0x5622c00f3ff0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b8bf8c740 10 client.0 _ll_register_callbacks cb 0x7f4b6c004a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00bb4b0 0x5622c00f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b82800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:55810/0 (socket says 172.21.15.2:55810) 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 -- 172.21.15.2:0/2801754606 learned_addr learned my addr 172.21.15.2:0/2801754606 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 -- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 msgr2=0x5622c00f7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 -- 172.21.15.2:0/2801754606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00bb4b0 msgr2=0x5622c00f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00bb4b0 0x5622c00f7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.574+0000 7f4b79400640 1 -- 172.21.15.2:0/2801754606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b740039b0 con 0x5622c00f3ff0 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b79400640 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0a2900 tx=0x7f4b6c09aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dea5b4cb278429a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b67e00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:37:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0cb640 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622c007fd30 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f4b6c09ad90 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b6c09a1d0 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b8bf8c740 10 client.25750 fetch_fsmap learned FSMap version 26 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5622c0058c50 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.575+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5622c007fd30 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 26) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4b6c0a9030 con 0x5622c00f3ff0 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 10 client.25750 fetch_fsmap finished waiting for FSMap version 26 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 10 client.25750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 20 client.25750 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c00fd570 0x5622c0167050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 4 client.25750 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:37:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5622bfd5a600 con 0x5622c00fd570 2024-03-11T18:37:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b81e00640 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c00fd570 0x5622c0167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b81e00640 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c00fd570 0x5622c0167050 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d639f6e79640e6d server_cookie=c68729810d525e6b in_seq=0 out_seq=0 2024-03-11T18:37:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.576+0000 7f4b67e00640 10 client.25750 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:06 smithi165 ceph-mon[36634]: pgmap v104: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-11T18:37:06.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:06 smithi023 ceph-mon[36406]: pgmap v104: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-11T18:37:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.745+0000 7f4b67e00640 1 -- 172.21.15.2:0/2801754606 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5622bfd5a600 con 0x5622c00fd570 2024-03-11T18:37:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.745+0000 7f4b67e00640 10 client.25750 handle_command_reply: tid=0 2024-03-11T18:37:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.745+0000 7f4b8bf8c740 1 client.25750 shutdown 2024-03-11T18:37:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.746+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 msgr2=0x5622c00f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.749 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.746+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0a2900 tx=0x7f4b6c09aba0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.750 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 shutdown_connections 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5622c00fd570 0x5622c0167050 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c00f4d60 0x5622c00f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5622c00f3ff0 0x5622c00f7940 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 --2- 172.21.15.2:0/2801754606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5622c00bb4b0 0x5622c00f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 >> 172.21.15.2:0/2801754606 conn(0x5622c00ffb40 msgr2=0x5622c00f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b67e00640 0 client.25750 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 shutdown_connections 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.748+0000 7f4b8bf8c740 1 -- 172.21.15.2:0/2801754606 wait complete. 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.749+0000 7f4b8bf8c740 20 client.25750 trim_cache size 0 max 16384 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:37:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 msgr2=0x7f4b680a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 0x7f4b680a40c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x5622c00f54c0 tx=0x7f4b74009500 comp rx=0 tx=0).stop 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 msgr2=0x7f4b841a3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00e9f0 tx=0x7f4b6c00edf0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 shutdown_connections 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4b680a1c00 0x7f4b680a40c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b84111f70 0x7f4b8419d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b84073e40 0x7f4b841a3d60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 --2- 172.21.15.2:0/1584316979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b84073520 0x7f4b841a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 >> 172.21.15.2:0/1584316979 conn(0x7f4b840fca30 msgr2=0x7f4b840fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 shutdown_connections 2024-03-11T18:37:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:06.750+0000 7f4b83200640 1 -- 172.21.15.2:0/1584316979 wait complete. 2024-03-11T18:37:07.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 215305 6.54 MB/sec execute 7 sec latency 106.045 ms 2024-03-11T18:37:08.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 216975 7.22 MB/sec execute 8 sec latency 10.415 ms 2024-03-11T18:37:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:08 smithi002 ceph-mon[31328]: pgmap v105: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-03-11T18:37:08.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:08 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:37:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:08 smithi023 ceph-mon[36406]: pgmap v105: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-03-11T18:37:08.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:08 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:37:08.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:08 smithi165 ceph-mon[36634]: pgmap v105: 129 pgs: 129 active+clean; 48 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-03-11T18:37:08.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:08 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:37:09.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 218513 7.05 MB/sec execute 9 sec latency 56.089 ms 2024-03-11T18:37:10.390 INFO:tasks.workunit.client.0.smithi002.stdout: 1 220403 7.68 MB/sec execute 10 sec latency 39.936 ms 2024-03-11T18:37:10.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:10 smithi002 ceph-mon[31328]: pgmap v106: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 386 op/s 2024-03-11T18:37:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:10 smithi165 ceph-mon[36634]: pgmap v106: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 386 op/s 2024-03-11T18:37:10.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:10 smithi023 ceph-mon[36406]: pgmap v106: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.1 MiB/s wr, 386 op/s 2024-03-11T18:37:11.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 222312 7.62 MB/sec execute 11 sec latency 19.852 ms 2024-03-11T18:37:12.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 224039 8.00 MB/sec execute 12 sec latency 19.202 ms 2024-03-11T18:37:12.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:12 smithi002 ceph-mon[31328]: pgmap v107: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:12.655 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:12 smithi023 ceph-mon[36406]: pgmap v107: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:12.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:12 smithi165 ceph-mon[36634]: pgmap v107: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:13.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 225883 7.92 MB/sec execute 13 sec latency 26.741 ms 2024-03-11T18:37:14.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 227384 8.20 MB/sec execute 14 sec latency 18.137 ms 2024-03-11T18:37:14.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:14 smithi002 ceph-mon[31328]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:14.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:14 smithi165 ceph-mon[36634]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:14.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:14 smithi023 ceph-mon[36406]: pgmap v108: 129 pgs: 129 active+clean; 45 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 291 op/s 2024-03-11T18:37:15.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 228219 7.81 MB/sec execute 15 sec latency 50.777 ms 2024-03-11T18:37:16.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 230257 8.03 MB/sec execute 16 sec latency 36.369 ms 2024-03-11T18:37:16.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:16 smithi002 ceph-mon[31328]: pgmap v109: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 398 op/s 2024-03-11T18:37:16.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:16 smithi165 ceph-mon[36634]: pgmap v109: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 398 op/s 2024-03-11T18:37:16.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:16 smithi023 ceph-mon[36406]: pgmap v109: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 398 op/s 2024-03-11T18:37:17.391 INFO:tasks.workunit.client.0.smithi002.stdout: 1 231339 7.91 MB/sec execute 17 sec latency 18.771 ms 2024-03-11T18:37:18.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 233034 7.85 MB/sec execute 18 sec latency 16.963 ms 2024-03-11T18:37:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:18 smithi002 ceph-mon[31328]: pgmap v110: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-11T18:37:18.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:18 smithi165 ceph-mon[36634]: pgmap v110: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-11T18:37:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:18 smithi023 ceph-mon[36406]: pgmap v110: 129 pgs: 129 active+clean; 52 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-11T18:37:19.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 234518 8.07 MB/sec execute 19 sec latency 19.025 ms 2024-03-11T18:37:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.965+0000 7f3ac6400640 1 -- 172.21.15.2:0/332825016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 msgr2=0x7f3ac010c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.965+0000 7f3ac6400640 1 --2- 172.21.15.2:0/332825016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac010c670 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800adb0 tx=0x7f3ab802cff0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.966+0000 7f3ac6400640 1 -- 172.21.15.2:0/332825016 shutdown_connections 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.966+0000 7f3ac6400640 1 --2- 172.21.15.2:0/332825016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac010c670 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.966+0000 7f3ac6400640 1 --2- 172.21.15.2:0/332825016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac00ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.966+0000 7f3ac6400640 1 --2- 172.21.15.2:0/332825016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ac01057b0 0x7f3ac0105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.966+0000 7f3ac6400640 1 -- 172.21.15.2:0/332825016 >> 172.21.15.2:0/332825016 conn(0x7f3ac00753a0 msgr2=0x7f3ac00757b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.967+0000 7f3ac6400640 1 -- 172.21.15.2:0/332825016 shutdown_connections 2024-03-11T18:37:19.969 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.967+0000 7f3ac6400640 1 -- 172.21.15.2:0/332825016 wait complete. 2024-03-11T18:37:19.972 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.970+0000 7f3ac6400640 1 Processor -- start 2024-03-11T18:37:19.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.973+0000 7f3ac6400640 1 -- start start 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.973+0000 7f3ac6400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.973+0000 7f3ac6400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.973+0000 7f3ac6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ac01057b0 0x7f3ac01997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.973+0000 7f3ac6400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0199f50 con 0x7f3ac01057b0 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3ac6400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac019a090 con 0x7f3ac01003e0 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3ac6400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac019a250 con 0x7f3ac00ffa20 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3abf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3abf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:60514/0 (socket says 172.21.15.2:60514) 2024-03-11T18:37:19.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3abf400640 1 -- 172.21.15.2:0/4113752223 learned_addr learned my addr 172.21.15.2:0/4113752223 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.974+0000 7f3abea00640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.975+0000 7f3abf400640 1 -- 172.21.15.2:0/4113752223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 msgr2=0x7f3ac019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.975+0000 7f3abf400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.975+0000 7f3abf400640 1 -- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ac01057b0 msgr2=0x7f3ac01997c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:37:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.975+0000 7f3abf400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ac01057b0 0x7f3ac01997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.975+0000 7f3abf400640 1 -- 172.21.15.2:0/4113752223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab800aa10 con 0x7f3ac00ffa20 2024-03-11T18:37:19.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.976+0000 7f3abf400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000d980 tx=0x7f3ab000de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.976+0000 7f3ab7e00640 1 -- 172.21.15.2:0/4113752223 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0009c30 con 0x7f3ac00ffa20 2024-03-11T18:37:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.976+0000 7f3ab7e00640 1 -- 172.21.15.2:0/4113752223 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ab000b660 con 0x7f3ac00ffa20 2024-03-11T18:37:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.976+0000 7f3ab7e00640 1 -- 172.21.15.2:0/4113752223 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab000f6b0 con 0x7f3ac00ffa20 2024-03-11T18:37:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.977+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac019a530 con 0x7f3ac00ffa20 2024-03-11T18:37:19.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.977+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac006ad80 con 0x7f3ac00ffa20 2024-03-11T18:37:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.982+0000 7f3ab7e00640 1 -- 172.21.15.2:0/4113752223 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f3ab0010460 con 0x7f3ac00ffa20 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 Processor -- start 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 -- start start 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b60870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b61230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9b1c620 con 0x55d4c9b61b20 2024-03-11T18:37:19.988 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9ace8c0 con 0x55d4c9b6a330 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.983+0000 7f3ac922e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9acbdf0 con 0x55d4c9b60db0 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3abf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b61230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3abea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3abea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38496/0 (socket says 172.21.15.2:38496) 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3abea00640 1 -- 172.21.15.2:0/1551757157 learned_addr learned my addr 172.21.15.2:0/1551757157 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3ab7e00640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 0x7f3aa40a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3abfe00640 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b60870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.984+0000 7f3ab7e00640 1 -- 172.21.15.2:0/4113752223 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3ab00cd7e0 con 0x7f3ac00ffa20 2024-03-11T18:37:19.989 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abfe00640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 0x7f3aa40a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 -- 172.21.15.2:0/1551757157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 msgr2=0x55d4c9b61230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b61230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 -- 172.21.15.2:0/1551757157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 msgr2=0x55d4c9b60870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b60870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 -- 172.21.15.2:0/1551757157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4c9a48b00 con 0x55d4c9b6a330 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abf400640 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b61230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abea00640 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00bc80 tx=0x7f3aac00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5abb3d800a9cbe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.985+0000 7f3abfe00640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 0x7f3aa40a4220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800a9e0 tx=0x7f3ab800a6d0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.986+0000 7f3ab4c00640 1 -- 172.21.15.2:0/1551757157 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac01a460 con 0x55d4c9b6a330 2024-03-11T18:37:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.986+0000 7f3ab4c00640 1 -- 172.21.15.2:0/1551757157 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3aac01aaa0 con 0x55d4c9b6a330 2024-03-11T18:37:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.986+0000 7f3ab4c00640 1 -- 172.21.15.2:0/1551757157 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac011b20 con 0x55d4c9b6a330 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 -- 172.21.15.2:0/1551757157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 msgr2=0x55d4c9b6c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00bc80 tx=0x7f3aac00f730 comp rx=0 tx=0).stop 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 -- 172.21.15.2:0/1551757157 shutdown_connections 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b6c720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b61230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 --2- 172.21.15.2:0/1551757157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b60870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 -- 172.21.15.2:0/1551757157 >> 172.21.15.2:0/1551757157 conn(0x55d4c9b284b0 msgr2=0x55d4c9b5baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 -- 172.21.15.2:0/1551757157 shutdown_connections 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.987+0000 7f3ac922e740 1 -- 172.21.15.2:0/1551757157 wait complete. 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.988+0000 7f3ac922e740 1 -- start start 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.988+0000 7f3ac922e740 1 Processor -- start 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b60db0 0x55d4c9b67370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b6a330 0x55d4c9b67df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9ac75d0 con 0x55d4c9b60db0 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9a6cb50 con 0x55d4c9b61b20 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9a6c7f0 con 0x55d4c9b6a330 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3ac922e740 10 client.0 _ll_register_callbacks cb 0x55d4c9a80550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38502/0 (socket says 172.21.15.2:38502) 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 -- 172.21.15.2:0/583632837 learned_addr learned my addr 172.21.15.2:0/583632837 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abf400640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b60db0 0x55d4c9b67370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:19.998 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 -- 172.21.15.2:0/583632837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b6a330 msgr2=0x55d4c9b67df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b6a330 0x55d4c9b67df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 -- 172.21.15.2:0/583632837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b60db0 msgr2=0x55d4c9b67370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b60db0 0x55d4c9b67370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.989+0000 7f3abea00640 1 -- 172.21.15.2:0/583632837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac00b8e0 con 0x55d4c9b61b20 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.990+0000 7f3abea00640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00b410 tx=0x7f3aac00bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b9bbb368bd25568 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.990+0000 7f3aa9600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.990+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac011cb0 con 0x55d4c9b61b20 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.990+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3aac011e10 con 0x55d4c9b61b20 2024-03-11T18:37:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.990+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4c9a99cd0 con 0x55d4c9b61b20 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3aac004020 con 0x55d4c9b61b20 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 10 client.25766 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d4c9ac5c50 con 0x55d4c9b61b20 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4c9a99cd0 con 0x55d4c9b61b20 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3aac022070 con 0x55d4c9b61b20 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 10 client.25766 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 10 client.25766 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 20 client.25766 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9b82ab0 0x55d4c9b84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 4 client.25766 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4c9abdce0 con 0x55d4c9b82ab0 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.991+0000 7f3abf400640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9b82ab0 0x55d4c9b84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.992+0000 7f3abf400640 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9b82ab0 0x55d4c9b84ef0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9632ebe579ad083 server_cookie=fb0d75d7f5356c17 in_seq=0 out_seq=0 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.992+0000 7f3aa9600640 10 client.25766 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3aa9600640 1 -- 172.21.15.2:0/583632837 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d4c9abdce0 con 0x55d4c9b82ab0 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3aa9600640 10 client.25766 handle_command_reply: tid=0 2024-03-11T18:37:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 client.25766 shutdown 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 msgr2=0x55d4c9b678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00b410 tx=0x7f3aac00bf80 comp rx=0 tx=0).stop 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 shutdown_connections 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9b82ab0 0x55d4c9b84ef0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b6a330 0x55d4c9b67df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b61b20 0x55d4c9b678b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 --2- 172.21.15.2:0/583632837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b60db0 0x55d4c9b67370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 >> 172.21.15.2:0/583632837 conn(0x55d4c9b284b0 msgr2=0x55d4c9b5d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.993+0000 7f3aa9600640 0 client.25766 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.994+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 shutdown_connections 2024-03-11T18:37:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.994+0000 7f3ac922e740 1 -- 172.21.15.2:0/583632837 wait complete. 2024-03-11T18:37:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:19.994+0000 7f3ac922e740 20 client.25766 trim_cache size 0 max 16384 2024-03-11T18:37:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.010+0000 7f3ac922e740 1 Processor -- start 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.010+0000 7f3ac922e740 1 -- start start 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b69aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b67c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c980fe60 con 0x55d4c9b61b20 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9a99cd0 con 0x55d4c9b6a330 2024-03-11T18:37:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3ac922e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9b4d6a0 con 0x55d4c9b60db0 2024-03-11T18:37:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3abfe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3abfe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38508/0 (socket says 172.21.15.2:38508) 2024-03-11T18:37:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3abfe00640 1 -- 172.21.15.2:0/143794864 learned_addr learned my addr 172.21.15.2:0/143794864 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3abea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b67c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.011+0000 7f3abf400640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b69aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 -- 172.21.15.2:0/143794864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 msgr2=0x55d4c9b69aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b69aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 -- 172.21.15.2:0/143794864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 msgr2=0x55d4c9b67c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b67c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 -- 172.21.15.2:0/143794864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4c9a62530 con 0x55d4c9b6a330 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abea00640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b67c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abf400640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b69aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3abfe00640 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8033530 tx=0x7f3ab800a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b5aea308e2039e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3aabe00640 1 -- 172.21.15.2:0/143794864 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8035440 con 0x55d4c9b6a330 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3aabe00640 1 -- 172.21.15.2:0/143794864 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ab80355a0 con 0x55d4c9b6a330 2024-03-11T18:37:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.012+0000 7f3aabe00640 1 -- 172.21.15.2:0/143794864 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8033ac0 con 0x55d4c9b6a330 2024-03-11T18:37:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 -- 172.21.15.2:0/143794864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 msgr2=0x55d4c9b79960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8033530 tx=0x7f3ab800a910 comp rx=0 tx=0).stop 2024-03-11T18:37:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 -- 172.21.15.2:0/143794864 shutdown_connections 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b79960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b67c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 --2- 172.21.15.2:0/143794864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b69aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.013+0000 7f3ac922e740 1 -- 172.21.15.2:0/143794864 >> 172.21.15.2:0/143794864 conn(0x55d4c9b284b0 msgr2=0x55d4c9b5b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.014+0000 7f3ac922e740 1 -- 172.21.15.2:0/143794864 shutdown_connections 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.014+0000 7f3ac922e740 1 -- 172.21.15.2:0/143794864 wait complete. 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.014+0000 7f3ac922e740 1 -- start start 2024-03-11T18:37:20.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 Processor -- start 2024-03-11T18:37:20.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b76a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b76f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9bd90b0 con 0x55d4c9b61b20 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9bd9220 con 0x55d4c9b6a330 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c9bd93e0 con 0x55d4c9b60db0 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.015+0000 7f3ac922e740 10 client.0 _ll_register_callbacks cb 0x7f3ab8038070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:60562/0 (socket says 172.21.15.2:60562) 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 -- 172.21.15.2:0/19497127 learned_addr learned my addr 172.21.15.2:0/19497127 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:20.019 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abfe00640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b76f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abea00640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b76a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 -- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 msgr2=0x55d4c9b76f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b76f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 -- 172.21.15.2:0/19497127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 msgr2=0x55d4c9b76a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b76a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abf400640 1 -- 172.21.15.2:0/19497127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac019a530 con 0x55d4c9b60db0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.016+0000 7f3abea00640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b76a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3abf400640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009610 tx=0x7f3ab00cda90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f052c3bbd22f5d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab00a27b0 con 0x55d4c9b60db0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4c9aecd30 con 0x55d4c9b60db0 2024-03-11T18:37:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3ab0098310 con 0x55d4c9b60db0 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ab0010710 con 0x55d4c9b60db0 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3ac922e740 10 client.25765 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d4c9ac5c50 con 0x55d4c9b60db0 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4c9aecd30 con 0x55d4c9b60db0 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.017+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3ab0096680 con 0x55d4c9b60db0 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 10 client.25765 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 10 client.25765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 20 client.25765 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9bdaa30 0x55d4c9bdce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 4 client.25765 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d4c97c7600 con 0x55d4c9bdaa30 2024-03-11T18:37:20.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3abea00640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9bdaa30 0x55d4c9bdce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:20.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3abea00640 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9bdaa30 0x55d4c9bdce50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f72ad5f461a1e2 server_cookie=a8a29365f06e87a7 in_seq=0 out_seq=0 2024-03-11T18:37:20.023 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.018+0000 7f3aa9600640 10 client.25765 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3aa9600640 1 -- 172.21.15.2:0/19497127 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d4c97c7600 con 0x55d4c9bdaa30 2024-03-11T18:37:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3aa9600640 10 client.25765 handle_command_reply: tid=0 2024-03-11T18:37:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 client.25765 shutdown 2024-03-11T18:37:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 msgr2=0x55d4c9b76500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009610 tx=0x7f3ab00cda90 comp rx=0 tx=0).stop 2024-03-11T18:37:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 shutdown_connections 2024-03-11T18:37:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4c9bdaa30 0x55d4c9bdce50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4c9b6a330 0x55d4c9b76f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.192+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4c9b61b20 0x55d4c9b76a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3ac922e740 1 --2- 172.21.15.2:0/19497127 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4c9b60db0 0x55d4c9b76500 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3aa9600640 0 client.25765 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 >> 172.21.15.2:0/19497127 conn(0x55d4c9b284b0 msgr2=0x55d4c9b65b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 shutdown_connections 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3ac922e740 1 -- 172.21.15.2:0/19497127 wait complete. 2024-03-11T18:37:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.193+0000 7f3ac922e740 20 client.25765 trim_cache size 0 max 16384 2024-03-11T18:37:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:37:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:37:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:37:20.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 msgr2=0x7f3aa40a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 0x7f3aa40a4220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800a9e0 tx=0x7f3ab800a6d0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 msgr2=0x7f3ac019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000d980 tx=0x7f3ab000de40 comp rx=0 tx=0).stop 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 shutdown_connections 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3aa40a1d60 0x7f3aa40a4220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ac01057b0 0x7f3ac01997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ac01003e0 0x7f3ac019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 --2- 172.21.15.2:0/4113752223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ac00ffa20 0x7f3ac019f6b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.194+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 >> 172.21.15.2:0/4113752223 conn(0x7f3ac00753a0 msgr2=0x7f3ac00fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.195+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 shutdown_connections 2024-03-11T18:37:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:20.195+0000 7f3ac6400640 1 -- 172.21.15.2:0/4113752223 wait complete. 2024-03-11T18:37:20.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 235302 7.77 MB/sec execute 20 sec latency 113.414 ms 2024-03-11T18:37:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:20 smithi165 ceph-mon[36634]: pgmap v111: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 370 op/s 2024-03-11T18:37:20.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:20 smithi023 ceph-mon[36406]: pgmap v111: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 370 op/s 2024-03-11T18:37:20.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:20 smithi002 ceph-mon[31328]: pgmap v111: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 370 op/s 2024-03-11T18:37:21.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 237921 8.20 MB/sec execute 21 sec latency 9.882 ms 2024-03-11T18:37:22.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 239215 8.07 MB/sec execute 22 sec latency 13.009 ms 2024-03-11T18:37:22.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:22 smithi165 ceph-mon[36634]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:22.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:22 smithi023 ceph-mon[36406]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:22.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:22 smithi002 ceph-mon[31328]: pgmap v112: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:23.392 INFO:tasks.workunit.client.0.smithi002.stdout: 1 241601 8.33 MB/sec execute 23 sec latency 10.776 ms 2024-03-11T18:37:23.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:23 smithi165 ceph-mon[36634]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:23.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:23 smithi023 ceph-mon[36406]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:23.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:23 smithi002 ceph-mon[31328]: pgmap v113: 129 pgs: 129 active+clean; 55 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.3 MiB/s wr, 194 op/s 2024-03-11T18:37:24.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 243108 8.21 MB/sec execute 24 sec latency 13.651 ms 2024-03-11T18:37:25.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 245146 8.43 MB/sec execute 25 sec latency 55.582 ms 2024-03-11T18:37:25.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:25 smithi023 ceph-mon[36406]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-11T18:37:25.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:25 smithi165 ceph-mon[36634]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-11T18:37:26.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:25 smithi002 ceph-mon[31328]: pgmap v114: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-11T18:37:26.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 246519 8.31 MB/sec execute 26 sec latency 17.461 ms 2024-03-11T18:37:27.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 248693 8.51 MB/sec execute 27 sec latency 9.479 ms 2024-03-11T18:37:27.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:27 smithi165 ceph-mon[36634]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-03-11T18:37:27.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:27 smithi023 ceph-mon[36406]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-03-11T18:37:28.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:27 smithi002 ceph-mon[31328]: pgmap v115: 129 pgs: 129 active+clean; 46 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-03-11T18:37:28.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 249997 8.39 MB/sec execute 28 sec latency 21.768 ms 2024-03-11T18:37:29.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 252186 8.57 MB/sec execute 29 sec latency 9.575 ms 2024-03-11T18:37:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:29 smithi165 ceph-mon[36634]: pgmap v116: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 353 op/s 2024-03-11T18:37:29.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:29 smithi023 ceph-mon[36406]: pgmap v116: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 353 op/s 2024-03-11T18:37:30.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:29 smithi002 ceph-mon[31328]: pgmap v116: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 353 op/s 2024-03-11T18:37:30.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 253385 8.47 MB/sec execute 30 sec latency 96.657 ms 2024-03-11T18:37:31.393 INFO:tasks.workunit.client.0.smithi002.stdout: 1 255728 8.64 MB/sec execute 31 sec latency 11.560 ms 2024-03-11T18:37:31.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:31 smithi023 ceph-mon[36406]: pgmap v117: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:31.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:31 smithi165 ceph-mon[36634]: pgmap v117: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:32.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:31 smithi002 ceph-mon[31328]: pgmap v117: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:32.394 INFO:tasks.workunit.client.0.smithi002.stdout: 1 256980 8.54 MB/sec execute 32 sec latency 18.366 ms 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 -- 172.21.15.2:0/312427433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30281035e0 msgr2=0x7f30281039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 --2- 172.21.15.2:0/312427433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30281035e0 0x7f30281039c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f301c009900 tx=0x7f301c02bc40 comp rx=0 tx=0).stop 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 -- 172.21.15.2:0/312427433 shutdown_connections 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 --2- 172.21.15.2:0/312427433 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281086b0 0x7f3028110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 --2- 172.21.15.2:0/312427433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3028103f90 0x7f3028107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.351 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 --2- 172.21.15.2:0/312427433 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30281035e0 0x7f30281039c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 -- 172.21.15.2:0/312427433 >> 172.21.15.2:0/312427433 conn(0x7f30280fee00 msgr2=0x7f3028101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 -- 172.21.15.2:0/312427433 shutdown_connections 2024-03-11T18:37:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.349+0000 7f302fe00640 1 -- 172.21.15.2:0/312427433 wait complete. 2024-03-11T18:37:33.352 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.350+0000 7f302fe00640 1 Processor -- start 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.350+0000 7f302fe00640 1 -- start start 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30281086b0 0x7f302819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 0x7f302819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819ee00 con 0x7f30281086b0 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819ef70 con 0x7f30281035e0 2024-03-11T18:37:33.353 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302fe00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302819f130 con 0x7f302819e410 2024-03-11T18:37:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302de00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 0x7f302819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.354 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60466/0 (socket says 172.21.15.2:60466) 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302d400640 1 -- 172.21.15.2:0/3523806412 learned_addr learned my addr 172.21.15.2:0/3523806412 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.351+0000 7f302ca00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30281086b0 0x7f302819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 -- 172.21.15.2:0/3523806412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 msgr2=0x7f302819e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 0x7f302819e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 -- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30281086b0 msgr2=0x7f302819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30281086b0 0x7f302819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 -- 172.21.15.2:0/3523806412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009560 con 0x7f30281035e0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302de00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 0x7f302819e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302d400640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f301c005940 tx=0x7f301c02c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f3026000640 1 -- 172.21.15.2:0/3523806412 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0042d0 con 0x7f30281035e0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30281a66c0 con 0x7f30281035e0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f3026000640 1 -- 172.21.15.2:0/3523806412 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f301c004430 con 0x7f30281035e0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.352+0000 7f3026000640 1 -- 172.21.15.2:0/3523806412 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c003c50 con 0x7f30281035e0 2024-03-11T18:37:33.355 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.353+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30281a6b70 con 0x7f30281035e0 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 Processor -- start 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 -- start start 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f3026000640 1 -- 172.21.15.2:0/3523806412 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f301c0054f0 con 0x7f30281035e0 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 0x55ebeaa27870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 0x55ebeaa28230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea9e3620 con 0x55ebeaa28b20 2024-03-11T18:37:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea9958c0 con 0x55ebeaa27db0 2024-03-11T18:37:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.356+0000 7f30329cd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea992df0 con 0x55ebeaa31330 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 0x55ebeaa28230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 0x55ebeaa27870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 0x55ebeaa28230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60468/0 (socket says 172.21.15.2:60468) 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302d400640 1 -- 172.21.15.2:0/1129443681 learned_addr learned my addr 172.21.15.2:0/1129443681 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:46814/0 (socket says 172.21.15.2:46814) 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 -- 172.21.15.2:0/1129443681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 msgr2=0x55ebeaa33720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 -- 172.21.15.2:0/1129443681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 msgr2=0x55ebeaa28230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 0x55ebeaa28230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302de00640 1 -- 172.21.15.2:0/1129443681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebea90fb00 con 0x55ebeaa28b20 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.357+0000 7f302ca00640 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f302de00640 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 0x55ebeaa27870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3020005fd0 tx=0x7f3020011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ac2be87c3f7553 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f301a600640 1 -- 172.21.15.2:0/1129443681 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020004d20 con 0x55ebeaa28b20 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f301a600640 1 -- 172.21.15.2:0/1129443681 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3020004e80 con 0x55ebeaa28b20 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f301a600640 1 -- 172.21.15.2:0/1129443681 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3020004480 con 0x55ebeaa28b20 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f3026000640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 0x7f30100a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f3026000640 1 -- 172.21.15.2:0/3523806412 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f301c0efbc0 con 0x7f30281035e0 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.358+0000 7f302d400640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 0x7f30100a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 -- 172.21.15.2:0/1129443681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 msgr2=0x55ebeaa27870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 0x55ebeaa27870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3020005fd0 tx=0x7f3020011bf0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 -- 172.21.15.2:0/1129443681 shutdown_connections 2024-03-11T18:37:33.362 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa33720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa27db0 0x55ebeaa28230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 --2- 172.21.15.2:0/1129443681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa28b20 0x55ebeaa27870 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 -- 172.21.15.2:0/1129443681 >> 172.21.15.2:0/1129443681 conn(0x55ebea9ef4b0 msgr2=0x55ebeaa22aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 -- 172.21.15.2:0/1129443681 shutdown_connections 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f30329cd740 1 -- 172.21.15.2:0/1129443681 wait complete. 2024-03-11T18:37:33.363 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.359+0000 7f302d400640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 0x7f30100a40d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f301c005ec0 tx=0x7f301c0f3630 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 -- start start 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 Processor -- start 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea98e5d0 con 0x55ebeaa27db0 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea933b50 con 0x55ebeaa31330 2024-03-11T18:37:33.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.360+0000 7f30329cd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea9337f0 con 0x55ebeaa28b20 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f30329cd740 10 client.0 _ll_register_callbacks cb 0x55ebea947550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:46820/0 (socket says 172.21.15.2:46820) 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 -- 172.21.15.2:0/2563906666 learned_addr learned my addr 172.21.15.2:0/2563906666 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302de00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 -- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa31330 msgr2=0x55ebeaa2ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 -- 172.21.15.2:0/2563906666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 msgr2=0x55ebeaa2e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.361+0000 7f302ca00640 1 -- 172.21.15.2:0/2563906666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3020011440 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.362+0000 7f302ca00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f302806a770 tx=0x7f3014012950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4831a3724526ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.362+0000 7f300ea00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.362+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014017070 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.362+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebea960cd0 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.362+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3014004320 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30140044e0 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 10 client.25777 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ebea98cc50 con 0x55ebeaa28b20 2024-03-11T18:37:33.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ebea960cd0 con 0x55ebeaa28b20 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3014011a50 con 0x55ebeaa28b20 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 10 client.25777 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 10 client.25777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 20 client.25777 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaa41400 0x55ebeaa43820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.363+0000 7f30329cd740 4 client.25777 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.364+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebea984ce0 con 0x55ebeaa41400 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.364+0000 7f302de00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaa41400 0x55ebeaa43820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.364+0000 7f302de00640 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaa41400 0x55ebeaa43820 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34b851b98f1d17b server_cookie=d0c1790f58e51f3 in_seq=0 out_seq=0 2024-03-11T18:37:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.364+0000 7f300ea00640 10 client.25777 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f300ea00640 1 -- 172.21.15.2:0/2563906666 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ebea984ce0 con 0x55ebeaa41400 2024-03-11T18:37:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f300ea00640 10 client.25777 handle_command_reply: tid=0 2024-03-11T18:37:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 client.25777 shutdown 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 msgr2=0x55ebeaa2e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f302806a770 tx=0x7f3014012950 comp rx=0 tx=0).stop 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 shutdown_connections 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaa41400 0x55ebeaa43820 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2e900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 --2- 172.21.15.2:0/2563906666 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f300ea00640 0 client.25777 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 >> 172.21.15.2:0/2563906666 conn(0x55ebea9ef4b0 msgr2=0x55ebeaa247f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 shutdown_connections 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 1 -- 172.21.15.2:0/2563906666 wait complete. 2024-03-11T18:37:33.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.366+0000 7f30329cd740 20 client.25777 trim_cache size 0 max 16384 2024-03-11T18:37:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.382+0000 7f30329cd740 1 Processor -- start 2024-03-11T18:37:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.382+0000 7f30329cd740 1 -- start start 2024-03-11T18:37:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa38180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa386c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa2e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebea6d6e60 con 0x55ebeaa27db0 2024-03-11T18:37:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebeaa23cd0 con 0x55ebeaa28b20 2024-03-11T18:37:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f30329cd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebeaa1e7d0 con 0x55ebeaa31330 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f302de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa38180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f302ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f302de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa38180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53212/0 (socket says 172.21.15.2:53212) 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f302de00640 1 -- 172.21.15.2:0/1852094994 learned_addr learned my addr 172.21.15.2:0/1852094994 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.383+0000 7f302de00640 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa2e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 -- 172.21.15.2:0/1852094994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 msgr2=0x55ebeaa2e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa2e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 -- 172.21.15.2:0/1852094994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 msgr2=0x55ebeaa38180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa38180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 -- 172.21.15.2:0/1852094994 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebea929530 con 0x55ebeaa28b20 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f302ca00640 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa386c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3014012010 tx=0x7f3014009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202a3929e77566c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f3019c00640 1 -- 172.21.15.2:0/1852094994 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014017070 con 0x55ebeaa28b20 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f3019c00640 1 -- 172.21.15.2:0/1852094994 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3014018040 con 0x55ebeaa28b20 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.384+0000 7f3019c00640 1 -- 172.21.15.2:0/1852094994 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301400fa20 con 0x55ebeaa28b20 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 -- 172.21.15.2:0/1852094994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 msgr2=0x55ebeaa386c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa386c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3014012010 tx=0x7f3014009240 comp rx=0 tx=0).stop 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 -- 172.21.15.2:0/1852094994 shutdown_connections 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa31330 0x55ebeaa2e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa386c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 --2- 172.21.15.2:0/1852094994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa27db0 0x55ebeaa38180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 -- 172.21.15.2:0/1852094994 >> 172.21.15.2:0/1852094994 conn(0x55ebea9ef4b0 msgr2=0x55ebeaa2cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 -- 172.21.15.2:0/1852094994 shutdown_connections 2024-03-11T18:37:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.385+0000 7f30329cd740 1 -- 172.21.15.2:0/1852094994 wait complete. 2024-03-11T18:37:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.386+0000 7f30329cd740 1 -- start start 2024-03-11T18:37:33.390 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.386+0000 7f30329cd740 1 Processor -- start 2024-03-11T18:37:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.386+0000 7f30329cd740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebeaaa0130 con 0x55ebeaa31330 2024-03-11T18:37:33.393 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebeaaa0270 con 0x55ebeaa28b20 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebeaaa03b0 con 0x55ebeaa27db0 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f30329cd740 10 client.0 _ll_register_callbacks cb 0x7f3014009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60492/0 (socket says 172.21.15.2:60492) 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 -- 172.21.15.2:0/736797992 learned_addr learned my addr 172.21.15.2:0/736797992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302de00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302de00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 -- 172.21.15.2:0/736797992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 msgr2=0x55ebeaa2afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 -- 172.21.15.2:0/736797992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa31330 msgr2=0x55ebeaa2ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 -- 172.21.15.2:0/736797992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebea98cc50 con 0x55ebeaa28b20 2024-03-11T18:37:33.395 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302de00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f302ca00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3014000c00 tx=0x7f3014005390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b4cfa774881f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.387+0000 7f300ea00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014017070 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3014000e40 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebea9b3d30 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3014005640 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f30329cd740 10 client.25790 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ebea94f2c0 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.388+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ebea9b3d30 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3014003710 con 0x55ebeaa28b20 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 10 client.25790 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 10 client.25790 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 20 client.25790 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaaa1630 0x55ebeaaa3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 4 client.25790 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:37:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ebea68e600 con 0x55ebeaaa1630 2024-03-11T18:37:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f302de00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaaa1630 0x55ebeaaa3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f302de00640 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaaa1630 0x55ebeaaa3a20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e4b08dd877326d server_cookie=61f406e7b0142223 in_seq=0 out_seq=0 2024-03-11T18:37:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.389+0000 7f300ea00640 10 client.25790 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:33.397 INFO:tasks.workunit.client.0.smithi002.stdout: 1 259342 8.70 MB/sec execute 33 sec latency 9.784 ms 2024-03-11T18:37:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.605+0000 7f300ea00640 1 -- 172.21.15.2:0/736797992 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ebea68e600 con 0x55ebeaaa1630 2024-03-11T18:37:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.605+0000 7f300ea00640 10 client.25790 handle_command_reply: tid=0 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 client.25790 shutdown 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 msgr2=0x55ebeaa2b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3014000c00 tx=0x7f3014005390 comp rx=0 tx=0).stop 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 shutdown_connections 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55ebeaaa1630 0x55ebeaaa3a20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ebeaa31330 0x55ebeaa2ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebeaa28b20 0x55ebeaa2b500 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 --2- 172.21.15.2:0/736797992 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55ebeaa27db0 0x55ebeaa2afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f300ea00640 0 client.25790 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 >> 172.21.15.2:0/736797992 conn(0x55ebea9ef4b0 msgr2=0x55ebeaa23e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 shutdown_connections 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 1 -- 172.21.15.2:0/736797992 wait complete. 2024-03-11T18:37:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.606+0000 7f30329cd740 20 client.25790 trim_cache size 0 max 16384 2024-03-11T18:37:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:37:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:37:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:37:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 msgr2=0x7f30100a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 0x7f30100a40d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f301c005ec0 tx=0x7f301c0f3630 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 msgr2=0x7f3028199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f301c005940 tx=0x7f301c02c680 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 shutdown_connections 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f302819e410 0x7f302819e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f30100a1c10 0x7f30100a40d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30281086b0 0x7f302819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 --2- 172.21.15.2:0/3523806412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30281035e0 0x7f3028199b20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:33.611 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 >> 172.21.15.2:0/3523806412 conn(0x7f30280fee00 msgr2=0x7f3028101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:33.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 shutdown_connections 2024-03-11T18:37:33.612 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:33.607+0000 7f302fe00640 1 -- 172.21.15.2:0/3523806412 wait complete. 2024-03-11T18:37:33.804 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:33 smithi002 ceph-mon[31328]: pgmap v118: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:33.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:33 smithi165 ceph-mon[36634]: pgmap v118: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:33 smithi023 ceph-mon[36406]: pgmap v118: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 2.0 MiB/s wr, 265 op/s 2024-03-11T18:37:34.394 INFO:tasks.workunit.client.0.smithi002.stdout: 1 260475 8.60 MB/sec execute 34 sec latency 95.635 ms 2024-03-11T18:37:35.394 INFO:tasks.workunit.client.0.smithi002.stdout: 1 262819 8.74 MB/sec execute 35 sec latency 21.412 ms 2024-03-11T18:37:35.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:35 smithi165 ceph-mon[36634]: pgmap v119: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 358 op/s 2024-03-11T18:37:35.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:35 smithi023 ceph-mon[36406]: pgmap v119: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 358 op/s 2024-03-11T18:37:36.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:35 smithi002 ceph-mon[31328]: pgmap v119: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 358 op/s 2024-03-11T18:37:36.394 INFO:tasks.workunit.client.0.smithi002.stdout: 1 264311 8.66 MB/sec execute 36 sec latency 34.847 ms 2024-03-11T18:37:37.394 INFO:tasks.workunit.client.0.smithi002.stdout: 1 266483 8.79 MB/sec execute 37 sec latency 9.693 ms 2024-03-11T18:37:37.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:37 smithi165 ceph-mon[36634]: pgmap v120: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:37:37.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:37 smithi023 ceph-mon[36406]: pgmap v120: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:37:38.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:37 smithi002 ceph-mon[31328]: pgmap v120: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.0 MiB/s wr, 183 op/s 2024-03-11T18:37:38.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 267836 8.70 MB/sec execute 38 sec latency 16.973 ms 2024-03-11T18:37:39.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 269890 8.83 MB/sec execute 39 sec latency 16.305 ms 2024-03-11T18:37:39.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:39 smithi165 ceph-mon[36634]: pgmap v121: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 373 op/s 2024-03-11T18:37:39.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:39 smithi023 ceph-mon[36406]: pgmap v121: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 373 op/s 2024-03-11T18:37:40.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:39 smithi002 ceph-mon[31328]: pgmap v121: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 373 op/s 2024-03-11T18:37:40.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 270790 8.66 MB/sec execute 40 sec latency 91.738 ms 2024-03-11T18:37:41.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 272978 8.76 MB/sec execute 41 sec latency 35.758 ms 2024-03-11T18:37:41.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:41 smithi165 ceph-mon[36634]: pgmap v122: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:41.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:41 smithi023 ceph-mon[36406]: pgmap v122: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:42.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:41 smithi002 ceph-mon[31328]: pgmap v122: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:42.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 274065 8.67 MB/sec execute 42 sec latency 15.638 ms 2024-03-11T18:37:43.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 276325 8.76 MB/sec execute 43 sec latency 14.357 ms 2024-03-11T18:37:43.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:43 smithi023 ceph-mon[36406]: pgmap v123: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:43.909 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:43 smithi165 ceph-mon[36634]: pgmap v123: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:44.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:43 smithi002 ceph-mon[31328]: pgmap v123: 129 pgs: 129 active+clean; 50 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 1.8 MiB/s wr, 283 op/s 2024-03-11T18:37:44.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 277483 8.71 MB/sec execute 44 sec latency 14.479 ms 2024-03-11T18:37:45.398 INFO:tasks.workunit.client.0.smithi002.stdout: 1 279301 8.68 MB/sec execute 45 sec latency 32.828 ms 2024-03-11T18:37:45.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:45 smithi165 ceph-mon[36634]: pgmap v124: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-11T18:37:45.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:45 smithi023 ceph-mon[36406]: pgmap v124: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-11T18:37:46.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:45 smithi002 ceph-mon[31328]: pgmap v124: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 376 op/s 2024-03-11T18:37:46.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 280778 8.74 MB/sec execute 46 sec latency 26.339 ms 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 -- 172.21.15.2:0/3877194747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c74112000 msgr2=0x7f0c741143f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 --2- 172.21.15.2:0/3877194747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c74112000 0x7f0c741143f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009cd0 tx=0x7f0c6c02c0e0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 -- 172.21.15.2:0/3877194747 shutdown_connections 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 --2- 172.21.15.2:0/3877194747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c74112000 0x7f0c741143f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 --2- 172.21.15.2:0/3877194747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 0x7f0c74111ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 --2- 172.21.15.2:0/3877194747 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c740756b0 0x7f0c74075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 -- 172.21.15.2:0/3877194747 >> 172.21.15.2:0/3877194747 conn(0x7f0c740fee70 msgr2=0x7f0c74101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 -- 172.21.15.2:0/3877194747 shutdown_connections 2024-03-11T18:37:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.775+0000 7f0c78e00640 1 -- 172.21.15.2:0/3877194747 wait complete. 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.776+0000 7f0c78e00640 1 Processor -- start 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.776+0000 7f0c78e00640 1 -- start start 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 0x7f0c7419e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c74112000 0x7f0c741a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741a2ce0 con 0x7f0c74075fd0 2024-03-11T18:37:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741a2e50 con 0x7f0c740756b0 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c78e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c741a3010 con 0x7f0c74112000 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56024/0 (socket says 172.21.15.2:56024) 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 -- 172.21.15.2:0/1044092393 learned_addr learned my addr 172.21.15.2:0/1044092393 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71400640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 0x7f0c7419e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c72800640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c74112000 0x7f0c741a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 -- 172.21.15.2:0/1044092393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c74112000 msgr2=0x7f0c741a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c74112000 0x7f0c741a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 -- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 msgr2=0x7f0c7419e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 0x7f0c7419e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.777+0000 7f0c71e00640 1 -- 172.21.15.2:0/1044092393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c009900 con 0x7f0c740756b0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c71e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400e9f0 tx=0x7f0c6400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c6aa00640 1 -- 172.21.15.2:0/1044092393 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64009720 con 0x7f0c740756b0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c6aa00640 1 -- 172.21.15.2:0/1044092393 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c6400cd80 con 0x7f0c740756b0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c741a32f0 con 0x7f0c740756b0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c6aa00640 1 -- 172.21.15.2:0/1044092393 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64010690 con 0x7f0c740756b0 2024-03-11T18:37:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.778+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741aab70 con 0x7f0c740756b0 2024-03-11T18:37:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.783+0000 7f0c7ba3a740 1 Processor -- start 2024-03-11T18:37:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.783+0000 7f0c7ba3a740 1 -- start start 2024-03-11T18:37:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.783+0000 7f0c6aa00640 1 -- 172.21.15.2:0/1044092393 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f0c640046c0 con 0x7f0c740756b0 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.783+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885aee720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885a9e620 con 0x55d885ae3b20 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885a508c0 con 0x55d885ae2db0 2024-03-11T18:37:46.786 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885a4ddf0 con 0x55d885aec330 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885aee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56038/0 (socket says 172.21.15.2:56038) 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 -- 172.21.15.2:0/2809111349 learned_addr learned my addr 172.21.15.2:0/2809111349 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c72800640 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 -- 172.21.15.2:0/2809111349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 msgr2=0x55d885aee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885aee720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 -- 172.21.15.2:0/2809111349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 msgr2=0x55d885ae2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71e00640 1 -- 172.21.15.2:0/2809111349 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8859cab00 con 0x55d885ae2db0 2024-03-11T18:37:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.784+0000 7f0c71400640 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885aee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c71e00640 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0c640cb200 tx=0x7f0c640d1e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ca885d4853976e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c6aa00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 0x7f0c580a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c5f400640 1 -- 172.21.15.2:0/2809111349 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64096a40 con 0x55d885ae2db0 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c5f400640 1 -- 172.21.15.2:0/2809111349 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c6409a570 con 0x55d885ae2db0 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c5f400640 1 -- 172.21.15.2:0/2809111349 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64069a50 con 0x55d885ae2db0 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.785+0000 7f0c6aa00640 1 -- 172.21.15.2:0/1044092393 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f0c64014070 con 0x7f0c740756b0 2024-03-11T18:37:46.789 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.786+0000 7f0c72800640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 0x7f0c580a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.786+0000 7f0c72800640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 0x7f0c580a4050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009ca0 tx=0x7f0c6c005c50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.786+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/2809111349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 msgr2=0x55d885ae3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.786+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0c640cb200 tx=0x7f0c640d1e10 comp rx=0 tx=0).stop 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/2809111349 shutdown_connections 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885aee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae2db0 0x55d885ae3230 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/2809111349 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/2809111349 >> 172.21.15.2:0/2809111349 conn(0x55d885aaa4b0 msgr2=0x55d885addaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/2809111349 shutdown_connections 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/2809111349 wait complete. 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 -- start start 2024-03-11T18:37:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.787+0000 7f0c7ba3a740 1 Processor -- start 2024-03-11T18:37:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885ae98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885ae9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885a495d0 con 0x55d885ae2db0 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8859eeb50 con 0x55d885ae3b20 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8859ee7f0 con 0x55d885aec330 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c7ba3a740 10 client.0 _ll_register_callbacks cb 0x55d885a02550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34850/0 (socket says 172.21.15.2:34850) 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c71e00640 1 -- 172.21.15.2:0/3074463832 learned_addr learned my addr 172.21.15.2:0/3074463832 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.788+0000 7f0c71400640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885ae98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c72800640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885ae9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 -- 172.21.15.2:0/3074463832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 msgr2=0x55d885ae9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885ae9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 -- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 msgr2=0x55d885ae98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885ae98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 -- 172.21.15.2:0/3074463832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c741aab70 con 0x55d885ae2db0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71400640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885ae98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c64097170 tx=0x7f0c64009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc73d72500979250 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c53e00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.789+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c640697f0 con 0x55d885ae2db0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f0c64069ed0 con 0x55d885ae2db0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d885a1bcd0 con 0x55d885ae2db0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c640d1390 con 0x55d885ae2db0 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c7ba3a740 10 client.15672 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d885a47c50 con 0x55d885ae2db0 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d885a1bcd0 con 0x55d885ae2db0 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c6401f060 con 0x55d885ae2db0 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c7ba3a740 10 client.15672 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.790+0000 7f0c7ba3a740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c7ba3a740 20 client.15672 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885afc010 0x55d885afe450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c7ba3a740 4 client.15672 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d885a3fce0 con 0x55d885afc010 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c71400640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885afc010 0x55d885afe450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c71400640 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885afc010 0x55d885afe450 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d977210ed27021a server_cookie=5dd67856df94b500 in_seq=0 out_seq=0 2024-03-11T18:37:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.791+0000 7f0c53e00640 10 client.15672 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c53e00640 1 -- 172.21.15.2:0/3074463832 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d885a3fce0 con 0x55d885afc010 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c53e00640 10 client.15672 handle_command_reply: tid=0 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 client.15672 shutdown 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 msgr2=0x55d885ae93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c64097170 tx=0x7f0c64009310 comp rx=0 tx=0).stop 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 shutdown_connections 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885afc010 0x55d885afe450 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885aec330 0x55d885ae9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885ae98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c53e00640 0 client.15672 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3074463832 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae2db0 0x55d885ae93a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.792+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 >> 172.21.15.2:0/3074463832 conn(0x55d885aaa4b0 msgr2=0x55d885adf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.793+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 shutdown_connections 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.793+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3074463832 wait complete. 2024-03-11T18:37:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.793+0000 7f0c7ba3a740 20 client.15672 trim_cache size 0 max 16384 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 Processor -- start 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 -- start start 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885aec330 0x55d885af3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885791e60 con 0x55d885ae3b20 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885adeaa0 con 0x55d885aec330 2024-03-11T18:37:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.809+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885ae2080 con 0x55d885ae2db0 2024-03-11T18:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:45714/0 (socket says 172.21.15.2:45714) 2024-03-11T18:37:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c72800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885aec330 0x55d885af3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 -- 172.21.15.2:0/499799054 learned_addr learned my addr 172.21.15.2:0/499799054 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 -- 172.21.15.2:0/499799054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885aec330 msgr2=0x55d885af3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885aec330 0x55d885af3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 -- 172.21.15.2:0/499799054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 msgr2=0x55d885ae9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 -- 172.21.15.2:0/499799054 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d885ae4fc0 con 0x55d885ae2db0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.810+0000 7f0c71e00640 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c64068040 tx=0x7f0c640d6870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72fb021a2ea3a7b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c5ea00640 1 -- 172.21.15.2:0/499799054 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6409a100 con 0x55d885ae2db0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c5ea00640 1 -- 172.21.15.2:0/499799054 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c64096a40 con 0x55d885ae2db0 2024-03-11T18:37:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c5ea00640 1 -- 172.21.15.2:0/499799054 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c640a2f40 con 0x55d885ae2db0 2024-03-11T18:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/499799054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 msgr2=0x55d885aebad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c64068040 tx=0x7f0c640d6870 comp rx=0 tx=0).stop 2024-03-11T18:37:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/499799054 shutdown_connections 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885aec330 0x55d885af3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885ae3b20 0x55d885ae9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/499799054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885aebad0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.811+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/499799054 >> 172.21.15.2:0/499799054 conn(0x55d885aaa4b0 msgr2=0x55d885ae7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.812+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/499799054 shutdown_connections 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.812+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/499799054 wait complete. 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.812+0000 7f0c7ba3a740 1 -- start start 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.812+0000 7f0c7ba3a740 1 Processor -- start 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885af3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 0x55d885aefca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885af01e0 con 0x55d885aec330 2024-03-11T18:37:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885af0350 con 0x55d885ae3b20 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d885af0510 con 0x55d885ae2db0 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c7ba3a740 10 client.0 _ll_register_callbacks cb 0x7f0c60005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:56068/0 (socket says 172.21.15.2:56068) 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 -- 172.21.15.2:0/3167189398 learned_addr learned my addr 172.21.15.2:0/3167189398 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c72800640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 0x55d885aefca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885af3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 -- 172.21.15.2:0/3167189398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 msgr2=0x55d885af3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885af3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 -- 172.21.15.2:0/3167189398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 msgr2=0x55d885aefca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 0x55d885aefca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c71400640 1 -- 172.21.15.2:0/3167189398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d885a47c50 con 0x55d885ae3b20 2024-03-11T18:37:46.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.813+0000 7f0c72800640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 0x55d885aefca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:37:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c71400640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0c60004700 tx=0x7f0c60013a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c0aaf7672e2b6f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:37:46.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c53e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60017070 con 0x55d885ae3b20 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f0c60005220 con 0x55d885ae3b20 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d885a6ed30 con 0x55d885ae3b20 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c600122e0 con 0x55d885ae3b20 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c7ba3a740 10 client.25808 fetch_fsmap learned FSMap version 27 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.814+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d8859e4530 con 0x55d885ae3b20 2024-03-11T18:37:46.819 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d885a6ed30 con 0x55d885ae3b20 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c60003660 con 0x55d885ae3b20 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 10 client.25808 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 10 client.25808 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 20 client.25808 populate_metadata read hostname 'smithi002' 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885b53b90 0x55d885b55fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 4 client.25808 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.815+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d885749600 con 0x55d885b53b90 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.816+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885b53b90 0x55d885b55fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.816+0000 7f0c71e00640 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885b53b90 0x55d885b55fb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46c2c5ff1fa007e4 server_cookie=922e5106f4dc6283 in_seq=0 out_seq=0 2024-03-11T18:37:46.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:46.816+0000 7f0c53e00640 10 client.25808 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.022+0000 7f0c53e00640 1 -- 172.21.15.2:0/3167189398 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d885749600 con 0x55d885b53b90 2024-03-11T18:37:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c53e00640 10 client.25808 handle_command_reply: tid=0 2024-03-11T18:37:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 client.25808 shutdown 2024-03-11T18:37:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 msgr2=0x55d885aef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0c60004700 tx=0x7f0c60013a70 comp rx=0 tx=0).stop 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 shutdown_connections 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d885b53b90 0x55d885b55fb0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d885aec330 0x55d885aefca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d885ae3b20 0x55d885aef760 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 --2- 172.21.15.2:0/3167189398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d885ae2db0 0x55d885af3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c53e00640 0 client.25808 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 >> 172.21.15.2:0/3167189398 conn(0x55d885aaa4b0 msgr2=0x55d885af83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.023+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 shutdown_connections 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.024+0000 7f0c7ba3a740 1 -- 172.21.15.2:0/3167189398 wait complete. 2024-03-11T18:37:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.024+0000 7f0c7ba3a740 20 client.25808 trim_cache size 0 max 16384 2024-03-11T18:37:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:37:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:37:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:37:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 msgr2=0x7f0c580a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 0x7f0c580a4050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c009ca0 tx=0x7f0c6c005c50 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 msgr2=0x7f0c7419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400e9f0 tx=0x7f0c6400edf0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 shutdown_connections 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c74112000 0x7f0c741a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f0c580a1b90 0x7f0c580a4050 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c74075fd0 0x7f0c7419e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 --2- 172.21.15.2:0/1044092393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c740756b0 0x7f0c7419de20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 >> 172.21.15.2:0/1044092393 conn(0x7f0c740fee70 msgr2=0x7f0c74113720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 shutdown_connections 2024-03-11T18:37:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:37:47.025+0000 7f0c78e00640 1 -- 172.21.15.2:0/1044092393 wait complete. 2024-03-11T18:37:47.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 282277 8.67 MB/sec execute 47 sec latency 143.613 ms 2024-03-11T18:37:47.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:47 smithi165 ceph-mon[36634]: pgmap v125: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:37:47.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:47 smithi023 ceph-mon[36406]: pgmap v125: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:37:48.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:47 smithi002 ceph-mon[31328]: pgmap v125: 129 pgs: 129 active+clean; 54 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 2.0 MiB/s wr, 283 op/s 2024-03-11T18:37:48.395 INFO:tasks.workunit.client.0.smithi002.stdout: 1 284032 8.76 MB/sec execute 48 sec latency 61.039 ms 2024-03-11T18:37:49.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 285194 8.69 MB/sec execute 49 sec latency 24.750 ms 2024-03-11T18:37:49.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:49 smithi165 ceph-mon[36634]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 470 op/s 2024-03-11T18:37:49.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:49 smithi023 ceph-mon[36406]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 470 op/s 2024-03-11T18:37:50.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:49 smithi002 ceph-mon[31328]: pgmap v126: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 470 op/s 2024-03-11T18:37:50.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 287297 8.77 MB/sec execute 50 sec latency 49.402 ms 2024-03-11T18:37:51.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 288235 8.64 MB/sec execute 51 sec latency 18.352 ms 2024-03-11T18:37:51.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:51 smithi023 ceph-mon[36406]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:51.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:51 smithi165 ceph-mon[36634]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:52.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:51 smithi002 ceph-mon[31328]: pgmap v127: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:52.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 290635 8.74 MB/sec execute 52 sec latency 21.739 ms 2024-03-11T18:37:53.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 291681 8.67 MB/sec execute 53 sec latency 13.802 ms 2024-03-11T18:37:53.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:53 smithi165 ceph-mon[36634]: pgmap v128: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:53.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:53 smithi023 ceph-mon[36406]: pgmap v128: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:54.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:53 smithi002 ceph-mon[31328]: pgmap v128: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 280 op/s 2024-03-11T18:37:54.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 293890 8.74 MB/sec execute 54 sec latency 14.753 ms 2024-03-11T18:37:55.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 295163 8.70 MB/sec execute 55 sec latency 60.677 ms 2024-03-11T18:37:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:55 smithi165 ceph-mon[36634]: pgmap v129: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-11T18:37:55.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:55 smithi023 ceph-mon[36406]: pgmap v129: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-11T18:37:56.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:55 smithi002 ceph-mon[31328]: pgmap v129: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-11T18:37:56.396 INFO:tasks.workunit.client.0.smithi002.stdout: 1 297325 8.75 MB/sec execute 56 sec latency 67.576 ms 2024-03-11T18:37:56.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:56 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:37:56.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:56 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:37:57.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:56 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:37:57.397 INFO:tasks.workunit.client.0.smithi002.stdout: 1 298684 8.73 MB/sec execute 57 sec latency 14.787 ms 2024-03-11T18:37:57.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:57 smithi165 ceph-mon[36634]: pgmap v130: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 282 op/s 2024-03-11T18:37:57.905 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:57 smithi023 ceph-mon[36406]: pgmap v130: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 282 op/s 2024-03-11T18:37:58.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:57 smithi002 ceph-mon[31328]: pgmap v130: 129 pgs: 129 active+clean; 52 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 282 op/s 2024-03-11T18:37:58.397 INFO:tasks.workunit.client.0.smithi002.stdout: 1 300903 8.79 MB/sec execute 58 sec latency 11.544 ms 2024-03-11T18:37:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:58 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:37:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:58 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:37:58.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:58 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:37:58.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:58 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:37:58.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:58 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:37:58.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:58 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:37:59.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:58 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:37:59.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:58 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:37:59.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:58 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:37:59.397 INFO:tasks.workunit.client.0.smithi002.stdout: 1 302330 8.76 MB/sec execute 59 sec latency 13.562 ms 2024-03-11T18:38:00.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:37:59 smithi165 ceph-mon[36634]: pgmap v131: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 383 op/s 2024-03-11T18:38:00.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:37:59 smithi023 ceph-mon[36406]: pgmap v131: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 383 op/s 2024-03-11T18:38:00.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 -- 172.21.15.2:0/13493322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae80ffab0 msgr2=0x7feae80fff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 --2- 172.21.15.2:0/13493322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae80ffab0 0x7feae80fff30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fead4009900 tx=0x7fead402bc40 comp rx=0 tx=0).stop 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 -- 172.21.15.2:0/13493322 shutdown_connections 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 --2- 172.21.15.2:0/13493322 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae8100470 0x7feae8110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 --2- 172.21.15.2:0/13493322 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae80ffab0 0x7feae80fff30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 --2- 172.21.15.2:0/13493322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 -- 172.21.15.2:0/13493322 >> 172.21.15.2:0/13493322 conn(0x7feae8075330 msgr2=0x7feae8075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 -- 172.21.15.2:0/13493322 shutdown_connections 2024-03-11T18:38:00.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.168+0000 7feaef000640 1 -- 172.21.15.2:0/13493322 wait complete. 2024-03-11T18:38:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.170+0000 7feaef000640 1 Processor -- start 2024-03-11T18:38:00.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.170+0000 7feaef000640 1 -- start start 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae8100470 0x7feae819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae8199ed0 con 0x7feae80ffab0 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae819a040 con 0x7feae8101f70 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaef000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae819a200 con 0x7feae8100470 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feae7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feae7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44808/0 (socket says 172.21.15.2:44808) 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feae7e00640 1 -- 172.21.15.2:0/2904870240 learned_addr learned my addr 172.21.15.2:0/2904870240 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:00.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feaed000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.175 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feae7400640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae8100470 0x7feae819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.171+0000 7feae7e00640 1 -- 172.21.15.2:0/2904870240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae8100470 msgr2=0x7feae819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae7e00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae8100470 0x7feae819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae7e00640 1 -- 172.21.15.2:0/2904870240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 msgr2=0x7feae8199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae7e00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae7e00640 1 -- 172.21.15.2:0/2904870240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4009560 con 0x7feae80ffab0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feaed000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae7e00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fead800e9f0 tx=0x7fead800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae4c00640 1 -- 172.21.15.2:0/2904870240 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8009720 con 0x7feae80ffab0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae4c00640 1 -- 172.21.15.2:0/2904870240 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fead800cd80 con 0x7feae80ffab0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feae4c00640 1 -- 172.21.15.2:0/2904870240 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8010690 con 0x7feae80ffab0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae819a4e0 con 0x7feae80ffab0 2024-03-11T18:38:00.176 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.172+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae81a6770 con 0x7feae80ffab0 2024-03-11T18:38:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.176+0000 7feaf1beb740 1 Processor -- start 2024-03-11T18:38:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.176+0000 7feae4c00640 1 -- 172.21.15.2:0/2904870240 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fead80046c0 con 0x7feae80ffab0 2024-03-11T18:38:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.176+0000 7feaf1beb740 1 -- start start 2024-03-11T18:38:00.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.176+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.176+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 0x563187cbd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cc6330 0x563187cc8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaf1beb740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187c78620 con 0x563187cc6330 2024-03-11T18:38:00.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaf1beb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187c2a8c0 con 0x563187cbdb20 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaf1beb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187c27df0 con 0x563187cbcdb0 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feae7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cc6330 0x563187cc8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:57178/0 (socket says 172.21.15.2:57178) 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 -- 172.21.15.2:0/3277632707 learned_addr learned my addr 172.21.15.2:0/3277632707 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feae7e00640 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 0x563187cbd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 -- 172.21.15.2:0/3277632707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 msgr2=0x563187cbd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 0x563187cbd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 -- 172.21.15.2:0/3277632707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cc6330 msgr2=0x563187cc8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cc6330 0x563187cc8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feaed000640 1 -- 172.21.15.2:0/3277632707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563187ba4b00 con 0x563187cbdb20 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.177+0000 7feae7e00640 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 0x563187cbd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:00.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.178+0000 7feaed000640 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feae00117e0 tx=0x7feae0011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daee7fed546af436 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.178+0000 7feadd600640 1 -- 172.21.15.2:0/3277632707 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0011310 con 0x563187cbdb20 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.178+0000 7feadd600640 1 -- 172.21.15.2:0/3277632707 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feae0003e50 con 0x563187cbdb20 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.178+0000 7feadd600640 1 -- 172.21.15.2:0/3277632707 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae0004590 con 0x563187cbdb20 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.178+0000 7feae4c00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 0x7feacc0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feae4c00640 1 -- 172.21.15.2:0/2904870240 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fead8014070 con 0x7feae80ffab0 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feae7e00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 0x7feacc0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 -- 172.21.15.2:0/3277632707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 msgr2=0x563187cbc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feae00117e0 tx=0x7feae0011ca0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 -- 172.21.15.2:0/3277632707 shutdown_connections 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cc6330 0x563187cc8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbcdb0 0x563187cbd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3277632707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbdb20 0x563187cbc870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 -- 172.21.15.2:0/3277632707 >> 172.21.15.2:0/3277632707 conn(0x563187c844b0 msgr2=0x563187cb7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 -- 172.21.15.2:0/3277632707 shutdown_connections 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feaf1beb740 1 -- 172.21.15.2:0/3277632707 wait complete. 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.179+0000 7feae7e00640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 0x7feacc0a4050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x563187cbe280 tx=0x7fead8004b30 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 -- start start 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 Processor -- start 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187cc3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187c235d0 con 0x563187cbcdb0 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187bc8b50 con 0x563187cc6330 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187bc87f0 con 0x563187cbdb20 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.180+0000 7feaf1beb740 10 client.0 _ll_register_callbacks cb 0x563187bdc550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feaed000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47410/0 (socket says 172.21.15.2:47410) 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 -- 172.21.15.2:0/2739403385 learned_addr learned my addr 172.21.15.2:0/2739403385 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feaed000640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187cc3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 -- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 msgr2=0x563187cc3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187cc3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 -- 172.21.15.2:0/2739403385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 msgr2=0x563187cc33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 -- 172.21.15.2:0/2739403385 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae00114c0 con 0x563187cbdb20 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feaed000640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187cc3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.181+0000 7feae7400640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fead4000c00 tx=0x7fead402ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3fe99302fff93e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7fead2000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead402cc10 con 0x563187cbdb20 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563187bf5cd0 con 0x563187cbdb20 2024-03-11T18:38:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fead4046070 con 0x563187cbdb20 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fead4004250 con 0x563187cbdb20 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7feaf1beb740 10 client.25813 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563187c21c50 con 0x563187cbdb20 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.182+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fead4053080 con 0x563187cbdb20 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fead403c040 con 0x563187cbdb20 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 10 client.25813 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 20 client.25813 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187cd6380 0x563187cd87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 4 client.25813 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563187c19ce0 con 0x563187cd6380 2024-03-11T18:38:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.183+0000 7feaed000640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187cd6380 0x563187cd87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7feaed000640 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187cd6380 0x563187cd87a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18e75bd316dddf30 server_cookie=54abc04a5d0d127c in_seq=0 out_seq=0 2024-03-11T18:38:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7fead2000640 10 client.25813 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7fead2000640 1 -- 172.21.15.2:0/2739403385 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563187c19ce0 con 0x563187cd6380 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7fead2000640 10 client.25813 handle_command_reply: tid=0 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7feaf1beb740 1 client.25813 shutdown 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.184+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 msgr2=0x563187cc3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fead4000c00 tx=0x7fead402ca20 comp rx=0 tx=0).stop 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 shutdown_connections 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187cd6380 0x563187cd87a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187cc3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187cc3900 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7fead2000640 0 client.25813 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 --2- 172.21.15.2:0/2739403385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 >> 172.21.15.2:0/2739403385 conn(0x563187c844b0 msgr2=0x563187cb97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 shutdown_connections 2024-03-11T18:38:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 1 -- 172.21.15.2:0/2739403385 wait complete. 2024-03-11T18:38:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.185+0000 7feaf1beb740 20 client.25813 trim_cache size 0 max 16384 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 Processor -- start 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 -- start start 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187ccd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187ccd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56318796be60 con 0x563187cbcdb0 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187cd2e90 con 0x563187cc6330 2024-03-11T18:38:00.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaf1beb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187cb3520 con 0x563187cbdb20 2024-03-11T18:38:00.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.200+0000 7feaed000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feaed000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44846/0 (socket says 172.21.15.2:44846) 2024-03-11T18:38:00.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187ccd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feaed000640 1 -- 172.21.15.2:0/972651299 learned_addr learned my addr 172.21.15.2:0/972651299 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:00.205 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feaed000640 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187ccd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 -- 172.21.15.2:0/972651299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 msgr2=0x563187ccd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187ccd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 -- 172.21.15.2:0/972651299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 msgr2=0x563187cc5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 -- 172.21.15.2:0/972651299 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563187bbe530 con 0x563187cbdb20 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feaed000640 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feae7400640 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187ccd150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fead4005ab0 tx=0x7fead4005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=108c27f080f3efa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.201+0000 7feadcc00640 1 -- 172.21.15.2:0/972651299 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4005850 con 0x563187cbdb20 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feadcc00640 1 -- 172.21.15.2:0/972651299 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563187bf5cd0 con 0x563187cbdb20 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feadcc00640 1 -- 172.21.15.2:0/972651299 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead4003e20 con 0x563187cbdb20 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 -- 172.21.15.2:0/972651299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 msgr2=0x563187ccd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187ccd150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fead4005ab0 tx=0x7fead4005e00 comp rx=0 tx=0).stop 2024-03-11T18:38:00.206 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 -- 172.21.15.2:0/972651299 shutdown_connections 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cc6330 0x563187ccd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cbdb20 0x563187ccd150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 --2- 172.21.15.2:0/972651299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbcdb0 0x563187cc5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 -- 172.21.15.2:0/972651299 >> 172.21.15.2:0/972651299 conn(0x563187c844b0 msgr2=0x563187cc1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 -- 172.21.15.2:0/972651299 shutdown_connections 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.202+0000 7feaf1beb740 1 -- 172.21.15.2:0/972651299 wait complete. 2024-03-11T18:38:00.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.203+0000 7feaf1beb740 1 -- start start 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.203+0000 7feaf1beb740 1 Processor -- start 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.203+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 0x563187cc0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 0x563187cc0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187d2ca60 con 0x563187cbdb20 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187d2cbd0 con 0x563187cbcdb0 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x563187d2cd90 con 0x563187cc6330 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaf1beb740 10 client.0 _ll_register_callbacks cb 0x7fead4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaed000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 0x563187cc0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaed000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 0x563187cc0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:57204/0 (socket says 172.21.15.2:57204) 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44848/0 (socket says 172.21.15.2:44848) 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaed000640 1 -- 172.21.15.2:0/3501860072 learned_addr learned my addr 172.21.15.2:0/3501860072 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaed000640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 0x563187cc0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 -- 172.21.15.2:0/3501860072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 msgr2=0x563187cc0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 0x563187cc0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 -- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 msgr2=0x563187cc0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 0x563187cc0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feae7400640 1 -- 172.21.15.2:0/3501860072 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563187ba4b00 con 0x563187cbdb20 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.204+0000 7feaed000640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 0x563187cc0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7feae7400640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x563187cb2e80 tx=0x7fead40036c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c7f9de6aab6ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7fead2000640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead40044a0 con 0x563187cbdb20 2024-03-11T18:38:00.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563187c48d30 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fead4046040 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fead4004250 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7feaf1beb740 10 client.15696 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.205+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563187c21c50 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fead4065080 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fead403c070 con 0x563187cbdb20 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 10 client.15696 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 20 client.15696 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187d2e100 0x563187d304f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 4 client.15696 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563187923600 con 0x563187d2e100 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.206+0000 7feaed000640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187d2e100 0x563187d304f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:00.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.207+0000 7feaed000640 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187d2e100 0x563187d304f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86161ef2943d5eae server_cookie=d17336ef006d956b in_seq=0 out_seq=0 2024-03-11T18:38:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.207+0000 7fead2000640 10 client.15696 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:00.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:37:59 smithi002 ceph-mon[31328]: pgmap v131: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 383 op/s 2024-03-11T18:38:00.398 INFO:tasks.workunit.client.0.smithi002.stdout: 1 303890 8.73 MB/sec execute 60 sec latency 258.650 ms 2024-03-11T18:38:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.603+0000 7fead2000640 1 -- 172.21.15.2:0/3501860072 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563187923600 con 0x563187d2e100 2024-03-11T18:38:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.603+0000 7fead2000640 10 client.15696 handle_command_reply: tid=0 2024-03-11T18:38:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.603+0000 7feaf1beb740 1 client.15696 shutdown 2024-03-11T18:38:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 msgr2=0x563187cc0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x563187cb2e80 tx=0x7fead40036c0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 shutdown_connections 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x563187d2e100 0x563187d304f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x563187cc6330 0x563187cc0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x563187cbdb20 0x563187cc0840 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7fead2000640 0 client.15696 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 --2- 172.21.15.2:0/3501860072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563187cbcdb0 0x563187cc0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 >> 172.21.15.2:0/3501860072 conn(0x563187c844b0 msgr2=0x563187cb64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 shutdown_connections 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 1 -- 172.21.15.2:0/3501860072 wait complete. 2024-03-11T18:38:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.604+0000 7feaf1beb740 20 client.15696 trim_cache size 0 max 16384 2024-03-11T18:38:00.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:38:00.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:38:00.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:38:00.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:38:00.609 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.605+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 msgr2=0x7feacc0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.605+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 0x7feacc0a4050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x563187cbe280 tx=0x7fead8004b30 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.605+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 msgr2=0x7feae819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.605+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fead800e9f0 tx=0x7fead800edf0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 shutdown_connections 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae8101f70 0x7feae8199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feae8100470 0x7feae819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7feacc0a1b90 0x7feacc0a4050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 --2- 172.21.15.2:0/2904870240 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae80ffab0 0x7feae819f6d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 >> 172.21.15.2:0/2904870240 conn(0x7feae8075330 msgr2=0x7feae80fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 shutdown_connections 2024-03-11T18:38:00.610 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:00.606+0000 7feaef000640 1 -- 172.21.15.2:0/2904870240 wait complete. 2024-03-11T18:38:01.398 INFO:tasks.workunit.client.0.smithi002.stdout: 1 304775 8.70 MB/sec execute 61 sec latency 98.460 ms 2024-03-11T18:38:02.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:01 smithi165 ceph-mon[36634]: pgmap v132: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:02.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:01 smithi023 ceph-mon[36406]: pgmap v132: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:01 smithi002 ceph-mon[31328]: pgmap v132: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:02.398 INFO:tasks.workunit.client.0.smithi002.stdout: 1 305299 8.64 MB/sec execute 62 sec latency 109.835 ms 2024-03-11T18:38:03.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 306590 8.58 MB/sec execute 63 sec latency 16.005 ms 2024-03-11T18:38:04.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:03 smithi165 ceph-mon[36634]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:03 smithi023 ceph-mon[36406]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:04.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:03 smithi002 ceph-mon[31328]: pgmap v133: 129 pgs: 129 active+clean; 56 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 195 op/s 2024-03-11T18:38:04.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 308617 8.64 MB/sec execute 64 sec latency 16.977 ms 2024-03-11T18:38:05.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 309487 8.54 MB/sec execute 65 sec latency 24.461 ms 2024-03-11T18:38:06.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:05 smithi023 ceph-mon[36406]: pgmap v134: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-11T18:38:06.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:05 smithi165 ceph-mon[36634]: pgmap v134: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-11T18:38:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:05 smithi002 ceph-mon[31328]: pgmap v134: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-11T18:38:06.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 311221 8.53 MB/sec execute 66 sec latency 55.743 ms 2024-03-11T18:38:07.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 312627 8.57 MB/sec execute 67 sec latency 22.099 ms 2024-03-11T18:38:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:07 smithi165 ceph-mon[36634]: pgmap v135: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 332 op/s 2024-03-11T18:38:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:07 smithi023 ceph-mon[36406]: pgmap v135: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 332 op/s 2024-03-11T18:38:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:07 smithi002 ceph-mon[31328]: pgmap v135: 129 pgs: 129 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 332 op/s 2024-03-11T18:38:08.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 313981 8.52 MB/sec execute 68 sec latency 18.609 ms 2024-03-11T18:38:09.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 315914 8.59 MB/sec execute 69 sec latency 14.724 ms 2024-03-11T18:38:10.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:09 smithi165 ceph-mon[36634]: pgmap v136: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-11T18:38:10.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:09 smithi023 ceph-mon[36406]: pgmap v136: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-11T18:38:10.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:09 smithi002 ceph-mon[31328]: pgmap v136: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-11T18:38:10.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 316942 8.50 MB/sec execute 70 sec latency 17.481 ms 2024-03-11T18:38:11.399 INFO:tasks.workunit.client.0.smithi002.stdout: 1 319222 8.60 MB/sec execute 71 sec latency 77.824 ms 2024-03-11T18:38:12.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:11 smithi165 ceph-mon[36634]: pgmap v137: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:12.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:11 smithi023 ceph-mon[36406]: pgmap v137: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:12.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:11 smithi002 ceph-mon[31328]: pgmap v137: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:12.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 320190 8.51 MB/sec execute 72 sec latency 18.105 ms 2024-03-11T18:38:13.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 322823 8.63 MB/sec execute 73 sec latency 12.545 ms 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.756+0000 7fb425800640 1 -- 172.21.15.2:0/3340190071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201033e0 msgr2=0x7fb4201037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.756+0000 7fb425800640 1 --2- 172.21.15.2:0/3340190071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201033e0 0x7fb4201037c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb410009900 tx=0x7fb41002bc40 comp rx=0 tx=0).stop 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 -- 172.21.15.2:0/3340190071 shutdown_connections 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 --2- 172.21.15.2:0/3340190071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201084b0 0x7fb420110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 --2- 172.21.15.2:0/3340190071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb420103d90 0x7fb420107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 --2- 172.21.15.2:0/3340190071 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201033e0 0x7fb4201037c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 -- 172.21.15.2:0/3340190071 >> 172.21.15.2:0/3340190071 conn(0x7fb4200fec60 msgr2=0x7fb420101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 -- 172.21.15.2:0/3340190071 shutdown_connections 2024-03-11T18:38:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.757+0000 7fb425800640 1 -- 172.21.15.2:0/3340190071 wait complete. 2024-03-11T18:38:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 Processor -- start 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 -- start start 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 0x7fb42019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201084b0 0x7fb42019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb420199fd0 con 0x7fb4201033e0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42019a140 con 0x7fb4201995e0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb425800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42019a300 con 0x7fb4201084b0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.758+0000 7fb41fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 0x7fb42019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb424e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb424e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:44470/0 (socket says 172.21.15.2:44470) 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb424e00640 1 -- 172.21.15.2:0/2061424652 learned_addr learned my addr 172.21.15.2:0/2061424652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb417e00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201084b0 0x7fb42019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 -- 172.21.15.2:0/2061424652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201084b0 msgr2=0x7fb42019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201084b0 0x7fb42019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 -- 172.21.15.2:0/2061424652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 msgr2=0x7fb420199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 -- 172.21.15.2:0/2061424652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb408009490 con 0x7fb4201033e0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb424e00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.759+0000 7fb41fe00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 0x7fb42019f430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb41002c180 tx=0x7fb41002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.760+0000 7fb41d600640 1 -- 172.21.15.2:0/2061424652 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4100042d0 con 0x7fb4201033e0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.760+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb410009560 con 0x7fb4201033e0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.760+0000 7fb41d600640 1 -- 172.21.15.2:0/2061424652 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb410004430 con 0x7fb4201033e0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.760+0000 7fb41d600640 1 -- 172.21.15.2:0/2061424652 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410003bb0 con 0x7fb4201033e0 2024-03-11T18:38:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.760+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4201a6630 con 0x7fb4201033e0 2024-03-11T18:38:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.763+0000 7fb41d600640 1 -- 172.21.15.2:0/2061424652 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fb410003d10 con 0x7fb4201033e0 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.763+0000 7fb429c2e740 1 Processor -- start 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 -- start start 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d29470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d32570 0x561a49d34960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49ce4620 con 0x561a49d32570 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c968c0 con 0x561a49d29d60 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb429c2e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c93df0 con 0x561a49d28ff0 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb424e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb41fe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d29470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb424e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:44478/0 (socket says 172.21.15.2:44478) 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.764+0000 7fb424e00640 1 -- 172.21.15.2:0/2322350366 learned_addr learned my addr 172.21.15.2:0/2322350366 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb417e00640 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d32570 0x561a49d34960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 -- 172.21.15.2:0/2322350366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 msgr2=0x561a49d29470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d29470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 -- 172.21.15.2:0/2322350366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d32570 msgr2=0x561a49d34960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d32570 0x561a49d34960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 -- 172.21.15.2:0/2322350366 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a49c10b00 con 0x561a49d29d60 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb41d600640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 0x7fb4040a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb41d600640 1 -- 172.21.15.2:0/2061424652 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb4100f00a0 con 0x7fb4201033e0 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb424e00640 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb418011850 tx=0x7fb418011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31ea1979b8b581c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb41fe00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 0x7fb4040a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.765+0000 7fb415600640 1 -- 172.21.15.2:0/2322350366 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418011330 con 0x561a49d29d60 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb415600640 1 -- 172.21.15.2:0/2322350366 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb41800fe80 con 0x561a49d29d60 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb415600640 1 -- 172.21.15.2:0/2322350366 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4180193e0 con 0x561a49d29d60 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 -- 172.21.15.2:0/2322350366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 msgr2=0x561a49d28ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb41fe00640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 0x7fb4040a4040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x561a49d2a4c0 tx=0x7fb4100f3b10 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb418011850 tx=0x7fb418011d10 comp rx=0 tx=0).stop 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 -- 172.21.15.2:0/2322350366 shutdown_connections 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d32570 0x561a49d34960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d29470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2322350366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d29d60 0x561a49d28ab0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 -- 172.21.15.2:0/2322350366 >> 172.21.15.2:0/2322350366 conn(0x561a49cf04b0 msgr2=0x561a49d23ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.766+0000 7fb429c2e740 1 -- 172.21.15.2:0/2322350366 shutdown_connections 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.767+0000 7fb429c2e740 1 -- 172.21.15.2:0/2322350366 wait complete. 2024-03-11T18:38:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.767+0000 7fb429c2e740 1 -- start start 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.767+0000 7fb429c2e740 1 Processor -- start 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d2f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c8f5d0 con 0x561a49d28ff0 2024-03-11T18:38:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c34b50 con 0x561a49d32570 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c347f0 con 0x561a49d29d60 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb429c2e740 10 client.0 _ll_register_callbacks cb 0x561a49c48550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb417e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb417e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:56872/0 (socket says 172.21.15.2:56872) 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb417e00640 1 -- 172.21.15.2:0/3138543464 learned_addr learned my addr 172.21.15.2:0/3138543464 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb424e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d2f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb424e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb417e00640 1 -- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 msgr2=0x561a49d2ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.768+0000 7fb417e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb417e00640 1 -- 172.21.15.2:0/3138543464 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 msgr2=0x561a49d2f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb417e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d2f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb417e00640 1 -- 172.21.15.2:0/3138543464 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb418011500 con 0x561a49d29d60 2024-03-11T18:38:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb424e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb417e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb4080126d0 tx=0x7fb408012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d37716bd1116cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb40e000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408017070 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a49c61cd0 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb408004050 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb408004210 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb429c2e740 10 client.25831 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.769+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561a49c8dc50 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561a49c61cd0 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb4080039c0 con 0x561a49d29d60 2024-03-11T18:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 10 client.25831 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 10 client.25831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 20 client.25831 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d42560 0x561a49d44980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 4 client.25831 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a49c85ce0 con 0x561a49d42560 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.770+0000 7fb424e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d42560 0x561a49d44980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.771+0000 7fb424e00640 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d42560 0x561a49d44980 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=706687f175359bd9 server_cookie=5a447b97a30be521 in_seq=0 out_seq=0 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.771+0000 7fb40e000640 10 client.25831 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb40e000640 1 -- 172.21.15.2:0/3138543464 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561a49c85ce0 con 0x561a49d42560 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb40e000640 10 client.25831 handle_command_reply: tid=0 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 client.25831 shutdown 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 msgr2=0x561a49d2fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb4080126d0 tx=0x7fb408012b90 comp rx=0 tx=0).stop 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 shutdown_connections 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d42560 0x561a49d44980 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d2fa80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 --2- 172.21.15.2:0/3138543464 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d2f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 >> 172.21.15.2:0/3138543464 conn(0x561a49cf04b0 msgr2=0x561a49d25a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb40e000640 0 client.25831 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 shutdown_connections 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 1 -- 172.21.15.2:0/3138543464 wait complete. 2024-03-11T18:38:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.772+0000 7fb429c2e740 20 client.25831 trim_cache size 0 max 16384 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 Processor -- start 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 -- start start 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d31c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a499d7e60 con 0x561a49d29d60 2024-03-11T18:38:13.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c85ce0 con 0x561a49d32570 2024-03-11T18:38:13.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.789+0000 7fb429c2e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49c61cd0 con 0x561a49d28ff0 2024-03-11T18:38:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb424e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d31c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55522/0 (socket says 172.21.15.2:55522) 2024-03-11T18:38:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb424e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d31c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:56884/0 (socket says 172.21.15.2:56884) 2024-03-11T18:38:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 -- 172.21.15.2:0/2157696741 learned_addr learned my addr 172.21.15.2:0/2157696741 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb424e00640 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 -- 172.21.15.2:0/2157696741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 msgr2=0x561a49d31c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d31c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 -- 172.21.15.2:0/2157696741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 msgr2=0x561a49d2e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.790+0000 7fb417e00640 1 -- 172.21.15.2:0/2157696741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a49c2a530 con 0x561a49d29d60 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.791+0000 7fb417e00640 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb408012420 tx=0x7fb40800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d7b31a7b01b4d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.791+0000 7fb414c00640 1 -- 172.21.15.2:0/2157696741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408017070 con 0x561a49d29d60 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.791+0000 7fb414c00640 1 -- 172.21.15.2:0/2157696741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb40800fe00 con 0x561a49d29d60 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.791+0000 7fb414c00640 1 -- 172.21.15.2:0/2157696741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4080050c0 con 0x561a49d29d60 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- 172.21.15.2:0/2157696741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 msgr2=0x561a49d2de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb408012420 tx=0x7fb40800f790 comp rx=0 tx=0).stop 2024-03-11T18:38:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- 172.21.15.2:0/2157696741 shutdown_connections 2024-03-11T18:38:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d2e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d29d60 0x561a49d2de10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 --2- 172.21.15.2:0/2157696741 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d28ff0 0x561a49d31c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- 172.21.15.2:0/2157696741 >> 172.21.15.2:0/2157696741 conn(0x561a49cf04b0 msgr2=0x561a49d23810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- 172.21.15.2:0/2157696741 shutdown_connections 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- 172.21.15.2:0/2157696741 wait complete. 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 -- start start 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.792+0000 7fb429c2e740 1 Processor -- start 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d37df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d38330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.798 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d38870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49d98a90 con 0x561a49d28ff0 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49d98bd0 con 0x561a49d32570 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a49d98d40 con 0x561a49d29d60 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb429c2e740 10 client.0 _ll_register_callbacks cb 0x7fb408009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb424e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d37df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb424e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d37df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55538/0 (socket says 172.21.15.2:55538) 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb417e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d38330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb424e00640 1 -- 172.21.15.2:0/1821921546 learned_addr learned my addr 172.21.15.2:0/1821921546 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.793+0000 7fb424e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d38870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 -- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 msgr2=0x561a49d38870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d38870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 -- 172.21.15.2:0/1821921546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 msgr2=0x561a49d37df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d37df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 -- 172.21.15.2:0/1821921546 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb408021040 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb417e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d38330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb408003d50 tx=0x7fb408009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ab908fa89167da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb40e000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408017030 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fb408018070 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a49cb4d30 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4080037a0 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb429c2e740 10 client.25837 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.794+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561a49c502c0 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561a49cb4d30 con 0x561a49d29d60 2024-03-11T18:38:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb408010760 con 0x561a49d29d60 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 10 client.25837 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 20 client.25837 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d9a180 0x561a49d9c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 4 client.25837 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561a4998f600 con 0x561a49d9a180 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.795+0000 7fb424e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d9a180 0x561a49d9c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.796+0000 7fb424e00640 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d9a180 0x561a49d9c5a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d67a4a66136f1ff9 server_cookie=f098d71cc245ce50 in_seq=0 out_seq=0 2024-03-11T18:38:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:13.796+0000 7fb40e000640 10 client.25837 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.051+0000 7fb40e000640 1 -- 172.21.15.2:0/1821921546 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561a4998f600 con 0x561a49d9a180 2024-03-11T18:38:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.051+0000 7fb40e000640 10 client.25837 handle_command_reply: tid=0 2024-03-11T18:38:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 client.25837 shutdown 2024-03-11T18:38:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 msgr2=0x561a49d38330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d38330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb408003d50 tx=0x7fb408009dc0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 shutdown_connections 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x561a49d9a180 0x561a49d9c5a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561a49d32570 0x561a49d38870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x561a49d29d60 0x561a49d38330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 --2- 172.21.15.2:0/1821921546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x561a49d28ff0 0x561a49d37df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 >> 172.21.15.2:0/1821921546 conn(0x561a49cf04b0 msgr2=0x561a49d2dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb40e000640 0 client.25837 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.052+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 shutdown_connections 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb429c2e740 1 -- 172.21.15.2:0/1821921546 wait complete. 2024-03-11T18:38:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb429c2e740 20 client.25837 trim_cache size 0 max 16384 2024-03-11T18:38:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:38:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:38:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:38:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 msgr2=0x7fb4040a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 0x7fb4040a4040 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x561a49d2a4c0 tx=0x7fb4100f3b10 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 msgr2=0x7fb42019f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.053+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 0x7fb42019f430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb41002c180 tx=0x7fb41002c680 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 shutdown_connections 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4201995e0 0x7fb420199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4201084b0 0x7fb42019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb4040a1b80 0x7fb4040a4040 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 --2- 172.21.15.2:0/2061424652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4201033e0 0x7fb42019f430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 >> 172.21.15.2:0/2061424652 conn(0x7fb4200fec60 msgr2=0x7fb420100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 shutdown_connections 2024-03-11T18:38:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:14.054+0000 7fb425800640 1 -- 172.21.15.2:0/2061424652 wait complete. 2024-03-11T18:38:14.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:13 smithi165 ceph-mon[36634]: pgmap v138: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:14.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:13 smithi023 ceph-mon[36406]: pgmap v138: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:14.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:13 smithi002 ceph-mon[31328]: pgmap v138: 129 pgs: 129 active+clean; 55 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.2 MiB/s wr, 319 op/s 2024-03-11T18:38:14.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 323798 8.54 MB/sec execute 74 sec latency 34.618 ms 2024-03-11T18:38:15.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 326062 8.61 MB/sec execute 75 sec latency 12.218 ms 2024-03-11T18:38:16.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:15 smithi165 ceph-mon[36634]: pgmap v139: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-03-11T18:38:16.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:15 smithi023 ceph-mon[36406]: pgmap v139: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-03-11T18:38:16.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:15 smithi002 ceph-mon[31328]: pgmap v139: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-03-11T18:38:16.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 326941 8.56 MB/sec execute 76 sec latency 58.240 ms 2024-03-11T18:38:17.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 328706 8.54 MB/sec execute 77 sec latency 36.760 ms 2024-03-11T18:38:18.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:17 smithi165 ceph-mon[36634]: pgmap v140: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.3 MiB/s wr, 177 op/s 2024-03-11T18:38:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:17 smithi023 ceph-mon[36406]: pgmap v140: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.3 MiB/s wr, 177 op/s 2024-03-11T18:38:18.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:17 smithi002 ceph-mon[31328]: pgmap v140: 129 pgs: 129 active+clean; 59 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 2.3 MiB/s wr, 177 op/s 2024-03-11T18:38:18.400 INFO:tasks.workunit.client.0.smithi002.stdout: 1 330281 8.58 MB/sec execute 78 sec latency 12.938 ms 2024-03-11T18:38:19.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 331534 8.54 MB/sec execute 79 sec latency 16.069 ms 2024-03-11T18:38:20.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:19 smithi165 ceph-mon[36634]: pgmap v141: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 394 op/s 2024-03-11T18:38:20.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:19 smithi023 ceph-mon[36406]: pgmap v141: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 394 op/s 2024-03-11T18:38:20.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:19 smithi002 ceph-mon[31328]: pgmap v141: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 394 op/s 2024-03-11T18:38:20.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 333664 8.60 MB/sec execute 80 sec latency 12.534 ms 2024-03-11T18:38:21.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 334554 8.53 MB/sec execute 81 sec latency 123.927 ms 2024-03-11T18:38:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:21 smithi165 ceph-mon[36634]: pgmap v142: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:22.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:21 smithi023 ceph-mon[36406]: pgmap v142: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:22.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:21 smithi002 ceph-mon[31328]: pgmap v142: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:22.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 336791 8.57 MB/sec execute 82 sec latency 37.538 ms 2024-03-11T18:38:23.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 337782 8.53 MB/sec execute 83 sec latency 17.135 ms 2024-03-11T18:38:24.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:23 smithi023 ceph-mon[36406]: pgmap v143: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:24.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:23 smithi165 ceph-mon[36634]: pgmap v143: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:24.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:23 smithi002 ceph-mon[31328]: pgmap v143: 129 pgs: 129 active+clean; 57 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.4 MiB/s wr, 306 op/s 2024-03-11T18:38:24.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 339754 8.56 MB/sec execute 84 sec latency 15.047 ms 2024-03-11T18:38:25.401 INFO:tasks.workunit.client.0.smithi002.stdout: 1 341130 8.55 MB/sec execute 85 sec latency 17.218 ms 2024-03-11T18:38:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:25 smithi023 ceph-mon[36406]: pgmap v144: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-03-11T18:38:26.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:25 smithi165 ceph-mon[36634]: pgmap v144: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-03-11T18:38:26.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:25 smithi002 ceph-mon[31328]: pgmap v144: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-03-11T18:38:26.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 342699 8.52 MB/sec execute 86 sec latency 53.496 ms 2024-03-11T18:38:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 -- 172.21.15.2:0/3983869250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba5806a450 msgr2=0x7fba58105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 --2- 172.21.15.2:0/3983869250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba5806a450 0x7fba58105e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fba44009930 tx=0x7fba4402bc30 comp rx=0 tx=0).stop 2024-03-11T18:38:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 -- 172.21.15.2:0/3983869250 shutdown_connections 2024-03-11T18:38:27.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 --2- 172.21.15.2:0/3983869250 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 0x7fba5810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 --2- 172.21.15.2:0/3983869250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba5806a450 0x7fba58105e10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 --2- 172.21.15.2:0/3983869250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba58069aa0 0x7fba58069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 -- 172.21.15.2:0/3983869250 >> 172.21.15.2:0/3983869250 conn(0x7fba58077170 msgr2=0x7fba58077580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 -- 172.21.15.2:0/3983869250 shutdown_connections 2024-03-11T18:38:27.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.190+0000 7fba5d200640 1 -- 172.21.15.2:0/3983869250 wait complete. 2024-03-11T18:38:27.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.191+0000 7fba5d200640 1 Processor -- start 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.191+0000 7fba5d200640 1 -- start start 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 0x7fba5819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba581975f0 0x7fba58197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba58197fe0 con 0x7fba581975f0 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba58198150 con 0x7fba58069aa0 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba5d200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba58198310 con 0x7fba581064f0 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42446/0 (socket says 172.21.15.2:42446) 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba55e00640 1 -- 172.21.15.2:0/2018415848 learned_addr learned my addr 172.21.15.2:0/2018415848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba55400640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 0x7fba5819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.192+0000 7fba56800640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba581975f0 0x7fba58197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 -- 172.21.15.2:0/2018415848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 msgr2=0x7fba5819d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 0x7fba5819d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 -- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba581975f0 msgr2=0x7fba58197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba581975f0 0x7fba58197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 -- 172.21.15.2:0/2018415848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44009560 con 0x7fba58069aa0 2024-03-11T18:38:27.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba55e00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba4ea00640 1 -- 172.21.15.2:0/2018415848 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48009700 con 0x7fba58069aa0 2024-03-11T18:38:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba4ea00640 1 -- 172.21.15.2:0/2018415848 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba4800cd80 con 0x7fba58069aa0 2024-03-11T18:38:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba4ea00640 1 -- 172.21.15.2:0/2018415848 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48018920 con 0x7fba58069aa0 2024-03-11T18:38:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.193+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba581a4300 con 0x7fba58069aa0 2024-03-11T18:38:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.194+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba581a4830 con 0x7fba58069aa0 2024-03-11T18:38:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 Processor -- start 2024-03-11T18:38:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 -- start start 2024-03-11T18:38:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba4ea00640 1 -- 172.21.15.2:0/2018415848 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fba48018a80 con 0x7fba58069aa0 2024-03-11T18:38:27.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 0x565531dd1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 0x565531dd2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531ddd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531d8d620 con 0x565531ddb330 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531d3f8c0 con 0x565531dd1db0 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531d3cdf0 con 0x565531dd2b20 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.197+0000 7fba55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 0x565531dd2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba55400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531ddd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba55e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 0x565531dd2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42452/0 (socket says 172.21.15.2:42452) 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba55e00640 1 -- 172.21.15.2:0/3433154789 learned_addr learned my addr 172.21.15.2:0/3433154789 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:27.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 0x565531dd1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 -- 172.21.15.2:0/3433154789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 msgr2=0x565531dd2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 0x565531dd2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 -- 172.21.15.2:0/3433154789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 msgr2=0x565531ddd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531ddd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 -- 172.21.15.2:0/3433154789 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565531cb9b00 con 0x565531dd2b20 2024-03-11T18:38:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.198+0000 7fba56800640 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 0x565531dd1870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba50009f00 tx=0x7fba50011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf589f3ee947c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba43400640 1 -- 172.21.15.2:0/3433154789 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50004d20 con 0x565531dd2b20 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba43400640 1 -- 172.21.15.2:0/3433154789 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba50004e80 con 0x565531dd2b20 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba43400640 1 -- 172.21.15.2:0/3433154789 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50005280 con 0x565531dd2b20 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba4ea00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 0x7fba3c0a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba4ea00640 1 -- 172.21.15.2:0/2018415848 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba48014070 con 0x7fba58069aa0 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3433154789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 msgr2=0x565531dd1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba55e00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 0x7fba3c0a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.199+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 0x565531dd1870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fba50009f00 tx=0x7fba50011810 comp rx=0 tx=0).stop 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3433154789 shutdown_connections 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531ddd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd1db0 0x565531dd2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3433154789 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd2b20 0x565531dd1870 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3433154789 >> 172.21.15.2:0/3433154789 conn(0x565531d994b0 msgr2=0x565531dccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3433154789 shutdown_connections 2024-03-11T18:38:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3433154789 wait complete. 2024-03-11T18:38:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba55e00640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 0x7fba3c0a40d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fba4809a4e0 tx=0x7fba480d17b0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 -- start start 2024-03-11T18:38:27.204 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.200+0000 7fba5fe6d740 1 Processor -- start 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531dd8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531d385d0 con 0x565531ddb330 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531cddb50 con 0x565531dd2b20 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531cdd7f0 con 0x565531dd1db0 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba5fe6d740 10 client.0 _ll_register_callbacks cb 0x565531cf1550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42458/0 (socket says 172.21.15.2:42458) 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba56800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531dd8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba55400640 1 -- 172.21.15.2:0/3146804489 learned_addr learned my addr 172.21.15.2:0/3146804489 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.201+0000 7fba56800640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 -- 172.21.15.2:0/3146804489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 msgr2=0x565531dd8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531dd8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 -- 172.21.15.2:0/3146804489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 msgr2=0x565531dd8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 -- 172.21.15.2:0/3146804489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba50009be0 con 0x565531dd2b20 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba56800640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531dd8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba55400640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fba44005ab0 tx=0x7fba4402ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce89834191edd9c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba37e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44004050 con 0x565531dd2b20 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565531d0acd0 con 0x565531dd2b20 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.202+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba440041b0 con 0x565531dd2b20 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba44004370 con 0x565531dd2b20 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 10 client.25853 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x565531d36c50 con 0x565531dd2b20 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x565531d0acd0 con 0x565531dd2b20 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fba4403c040 con 0x565531dd2b20 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 10 client.25853 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 10 client.25853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 20 client.25853 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531def7d0 0x565531df1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 4 client.25853 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.203+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565531d2ece0 con 0x565531def7d0 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.204+0000 7fba56800640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531def7d0 0x565531df1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.204+0000 7fba56800640 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531def7d0 0x565531df1bf0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=518e5aa46b89bb4e server_cookie=9648c4a4adabac65 in_seq=0 out_seq=0 2024-03-11T18:38:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.204+0000 7fba37e00640 10 client.25853 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba37e00640 1 -- 172.21.15.2:0/3146804489 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565531d2ece0 con 0x565531def7d0 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba37e00640 10 client.25853 handle_command_reply: tid=0 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 client.25853 shutdown 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 msgr2=0x565531dd88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fba44005ab0 tx=0x7fba4402ca10 comp rx=0 tx=0).stop 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 shutdown_connections 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531def7d0 0x565531df1bf0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd88d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/3146804489 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531dd8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba37e00640 0 client.25853 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 >> 172.21.15.2:0/3146804489 conn(0x565531d994b0 msgr2=0x565531dce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 shutdown_connections 2024-03-11T18:38:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 1 -- 172.21.15.2:0/3146804489 wait complete. 2024-03-11T18:38:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.206+0000 7fba5fe6d740 20 client.25853 trim_cache size 0 max 16384 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 Processor -- start 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 -- start start 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531ddaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531a80e60 con 0x565531ddb330 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531dcd8d0 con 0x565531dd2b20 2024-03-11T18:38:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531dbe6a0 con 0x565531dd1db0 2024-03-11T18:38:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba56800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531ddaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.221+0000 7fba56800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531ddaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:46684/0 (socket says 172.21.15.2:46684) 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba56800640 1 -- 172.21.15.2:0/562619156 learned_addr learned my addr 172.21.15.2:0/562619156 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42468/0 (socket says 172.21.15.2:42468) 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba56800640 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 -- 172.21.15.2:0/562619156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 msgr2=0x565531ddaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531ddaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 -- 172.21.15.2:0/562619156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 msgr2=0x565531dd6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 -- 172.21.15.2:0/562619156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565531cd3530 con 0x565531dd2b20 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.222+0000 7fba55400640 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fba4402c170 tx=0x7fba4402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5430ca9b4e69c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba42a00640 1 -- 172.21.15.2:0/562619156 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44003bf0 con 0x565531dd2b20 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba42a00640 1 -- 172.21.15.2:0/562619156 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba440035f0 con 0x565531dd2b20 2024-03-11T18:38:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba42a00640 1 -- 172.21.15.2:0/562619156 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba440052b0 con 0x565531dd2b20 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 -- 172.21.15.2:0/562619156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 msgr2=0x565531dd8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fba4402c170 tx=0x7fba4402bc30 comp rx=0 tx=0).stop 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 -- 172.21.15.2:0/562619156 shutdown_connections 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531ddb330 0x565531dd6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531dd8c30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.223+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/562619156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531dd1db0 0x565531ddaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.224+0000 7fba5fe6d740 1 -- 172.21.15.2:0/562619156 >> 172.21.15.2:0/562619156 conn(0x565531d994b0 msgr2=0x565531df6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.224+0000 7fba5fe6d740 1 -- 172.21.15.2:0/562619156 shutdown_connections 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.224+0000 7fba5fe6d740 1 -- 172.21.15.2:0/562619156 wait complete. 2024-03-11T18:38:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.224+0000 7fba5fe6d740 1 -- start start 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.224+0000 7fba5fe6d740 1 Processor -- start 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531dd1db0 0x565531de4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531ddb330 0x565531de5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531de65f0 con 0x565531dd1db0 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531de6760 con 0x565531dd2b20 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x565531de6920 con 0x565531ddb330 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba5fe6d740 10 client.0 _ll_register_callbacks cb 0x7fba44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:42474/0 (socket says 172.21.15.2:42474) 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 -- 172.21.15.2:0/2429377430 learned_addr learned my addr 172.21.15.2:0/2429377430 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba56800640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531dd1db0 0x565531de4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba56800640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531ddb330 0x565531de5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 -- 172.21.15.2:0/2429377430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531ddb330 msgr2=0x565531de5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531ddb330 0x565531de5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 -- 172.21.15.2:0/2429377430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531dd1db0 msgr2=0x565531de4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531dd1db0 0x565531de4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.225+0000 7fba55400640 1 -- 172.21.15.2:0/2429377430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565531d36c50 con 0x565531dd2b20 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba55400640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba44009850 tx=0x7fba4402c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb9526a28be626ab server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba37e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44004000 con 0x565531dd2b20 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565531d5dd30 con 0x565531dd2b20 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fba44000e40 con 0x565531dd2b20 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba44005670 con 0x565531dd2b20 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba5fe6d740 10 client.25859 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.226+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x565531cf92c0 con 0x565531dd2b20 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x565531d5dd30 con 0x565531dd2b20 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fba4403c040 con 0x565531dd2b20 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 10 client.25859 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 20 client.25859 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531e46bd0 0x565531e48ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 4 client.25859 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565531a38600 con 0x565531e46bd0 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.227+0000 7fba56800640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531e46bd0 0x565531e48ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.228+0000 7fba56800640 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531e46bd0 0x565531e48ff0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbff053f776a6498 server_cookie=4bd9d8d866f0cc7c in_seq=0 out_seq=0 2024-03-11T18:38:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.228+0000 7fba37e00640 10 client.25859 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:27.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 344368 8.57 MB/sec execute 87 sec latency 83.885 ms 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba37e00640 1 -- 172.21.15.2:0/2429377430 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565531a38600 con 0x565531e46bd0 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba37e00640 10 client.25859 handle_command_reply: tid=0 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 client.25859 shutdown 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 msgr2=0x565531de5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba44009850 tx=0x7fba4402c5d0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 shutdown_connections 2024-03-11T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x565531e46bd0 0x565531e48ff0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x565531ddb330 0x565531de5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x565531dd2b20 0x565531de5170 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 --2- 172.21.15.2:0/2429377430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565531dd1db0 0x565531de4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 >> 172.21.15.2:0/2429377430 conn(0x565531d994b0 msgr2=0x565531dcb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.429+0000 7fba37e00640 0 client.25859 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.430+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 shutdown_connections 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.430+0000 7fba5fe6d740 1 -- 172.21.15.2:0/2429377430 wait complete. 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.430+0000 7fba5fe6d740 20 client.25859 trim_cache size 0 max 16384 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:38:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 msgr2=0x7fba3c0a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 0x7fba3c0a40d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fba4809a4e0 tx=0x7fba480d17b0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 msgr2=0x7fba5819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fba4800e9f0 tx=0x7fba4800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 shutdown_connections 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fba3c0a1c10 0x7fba3c0a40d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba581975f0 0x7fba58197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba581064f0 0x7fba5819d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 --2- 172.21.15.2:0/2018415848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba58069aa0 0x7fba5819d440 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 >> 172.21.15.2:0/2018415848 conn(0x7fba58077170 msgr2=0x7fba580fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 shutdown_connections 2024-03-11T18:38:27.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:27.431+0000 7fba5d200640 1 -- 172.21.15.2:0/2018415848 wait complete. 2024-03-11T18:38:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:27 smithi023 ceph-mon[36406]: pgmap v145: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-03-11T18:38:28.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:27 smithi165 ceph-mon[36634]: pgmap v145: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-03-11T18:38:28.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:27 smithi002 ceph-mon[31328]: pgmap v145: 129 pgs: 129 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-03-11T18:38:28.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 345705 8.54 MB/sec execute 88 sec latency 20.135 ms 2024-03-11T18:38:29.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 347889 8.59 MB/sec execute 89 sec latency 10.268 ms 2024-03-11T18:38:30.155 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:29 smithi023 ceph-mon[36406]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 493 op/s 2024-03-11T18:38:30.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:29 smithi165 ceph-mon[36634]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 493 op/s 2024-03-11T18:38:30.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:29 smithi002 ceph-mon[31328]: pgmap v146: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 493 op/s 2024-03-11T18:38:30.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 349207 8.56 MB/sec execute 90 sec latency 16.173 ms 2024-03-11T18:38:31.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 351313 8.60 MB/sec execute 91 sec latency 51.935 ms 2024-03-11T18:38:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:31 smithi165 ceph-mon[36634]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:31 smithi023 ceph-mon[36406]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:31 smithi002 ceph-mon[31328]: pgmap v147: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:32.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 352137 8.53 MB/sec execute 92 sec latency 37.552 ms 2024-03-11T18:38:33.402 INFO:tasks.workunit.client.0.smithi002.stdout: 1 354156 8.57 MB/sec execute 93 sec latency 66.074 ms 2024-03-11T18:38:34.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:33 smithi165 ceph-mon[36634]: pgmap v148: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:34.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:33 smithi023 ceph-mon[36406]: pgmap v148: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:33 smithi002 ceph-mon[31328]: pgmap v148: 129 pgs: 129 active+clean; 53 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.0 MiB/s wr, 275 op/s 2024-03-11T18:38:34.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 355422 8.55 MB/sec execute 94 sec latency 14.649 ms 2024-03-11T18:38:35.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 357179 8.54 MB/sec execute 95 sec latency 16.971 ms 2024-03-11T18:38:36.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:35 smithi165 ceph-mon[36634]: pgmap v149: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 361 op/s 2024-03-11T18:38:36.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:35 smithi002 ceph-mon[31328]: pgmap v149: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 361 op/s 2024-03-11T18:38:36.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 358680 8.57 MB/sec execute 96 sec latency 30.983 ms 2024-03-11T18:38:36.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:35 smithi023 ceph-mon[36406]: pgmap v149: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 361 op/s 2024-03-11T18:38:37.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 360078 8.53 MB/sec execute 97 sec latency 51.478 ms 2024-03-11T18:38:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:37 smithi002 ceph-mon[31328]: pgmap v150: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:38:38.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 362079 8.58 MB/sec execute 98 sec latency 12.793 ms 2024-03-11T18:38:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:37 smithi165 ceph-mon[36634]: pgmap v150: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:38:38.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:37 smithi023 ceph-mon[36406]: pgmap v150: 129 pgs: 129 active+clean; 57 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:38:39.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 363175 8.55 MB/sec execute 99 sec latency 19.304 ms 2024-03-11T18:38:40.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:39 smithi002 ceph-mon[31328]: pgmap v151: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-11T18:38:40.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 365342 8.59 MB/sec execute 100 sec latency 12.240 ms 2024-03-11T18:38:40.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:39 smithi023 ceph-mon[36406]: pgmap v151: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-11T18:38:40.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:39 smithi165 ceph-mon[36634]: pgmap v151: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-03-11T18:38:40.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 -- 172.21.15.2:0/3605098112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c103f90 msgr2=0x7fa36c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 --2- 172.21.15.2:0/3605098112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c103f90 0x7fa36c107fd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa358009940 tx=0x7fa35802bc20 comp rx=0 tx=0).stop 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 -- 172.21.15.2:0/3605098112 shutdown_connections 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 --2- 172.21.15.2:0/3605098112 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1086b0 0x7fa36c110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 --2- 172.21.15.2:0/3605098112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c103f90 0x7fa36c107fd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 --2- 172.21.15.2:0/3605098112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1035e0 0x7fa36c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 -- 172.21.15.2:0/3605098112 >> 172.21.15.2:0/3605098112 conn(0x7fa36c0fee00 msgr2=0x7fa36c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 -- 172.21.15.2:0/3605098112 shutdown_connections 2024-03-11T18:38:40.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.583+0000 7fa372400640 1 -- 172.21.15.2:0/3605098112 wait complete. 2024-03-11T18:38:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.584+0000 7fa372400640 1 Processor -- start 2024-03-11T18:38:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.584+0000 7fa372400640 1 -- start start 2024-03-11T18:38:40.589 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1035e0 0x7fa36c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c199850 0x7fa36c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c19a300 con 0x7fa36c1035e0 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c19a470 con 0x7fa36c199850 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa372400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c19a5e0 con 0x7fa36c1086b0 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa36b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1035e0 0x7fa36c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa36aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa36aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:35186/0 (socket says 172.21.15.2:35186) 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa36aa00640 1 -- 172.21.15.2:0/3092472439 learned_addr learned my addr 172.21.15.2:0/3092472439 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.585+0000 7fa36be00640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c199850 0x7fa36c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 -- 172.21.15.2:0/3092472439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c199850 msgr2=0x7fa36c199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c199850 0x7fa36c199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.590 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 -- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1035e0 msgr2=0x7fa36c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1035e0 0x7fa36c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 -- 172.21.15.2:0/3092472439 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009490 con 0x7fa36c1086b0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa36aa00640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa358004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa363e00640 1 -- 172.21.15.2:0/3092472439 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa358005460 con 0x7fa36c1086b0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358009500 con 0x7fa36c1086b0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa363e00640 1 -- 172.21.15.2:0/3092472439 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3580055c0 con 0x7fa36c1086b0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa363e00640 1 -- 172.21.15.2:0/3092472439 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3580046c0 con 0x7fa36c1086b0 2024-03-11T18:38:40.591 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.586+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c1a6850 con 0x7fa36c1086b0 2024-03-11T18:38:40.592 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.590+0000 7fa363e00640 1 -- 172.21.15.2:0/3092472439 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fa35803b040 con 0x7fa36c1086b0 2024-03-11T18:38:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.590+0000 7fa375235740 1 Processor -- start 2024-03-11T18:38:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 -- start start 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59ddb0 0x557bff59e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff5a7330 0x557bff5a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff559620 con 0x557bff5a7330 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff50b8c0 con 0x557bff59eb20 2024-03-11T18:38:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa375235740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff508df0 con 0x557bff59ddb0 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa36be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa36b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59ddb0 0x557bff59e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa36be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37110/0 (socket says 172.21.15.2:37110) 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.591+0000 7fa36be00640 1 -- 172.21.15.2:0/3590735708 learned_addr learned my addr 172.21.15.2:0/3590735708 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36aa00640 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff5a7330 0x557bff5a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 -- 172.21.15.2:0/3590735708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59ddb0 msgr2=0x557bff59e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59ddb0 0x557bff59e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 -- 172.21.15.2:0/3590735708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff5a7330 msgr2=0x557bff5a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff5a7330 0x557bff5a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 -- 172.21.15.2:0/3590735708 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bff485b00 con 0x557bff59eb20 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa363e00640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 0x7fa3500a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36be00640 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa364009f50 tx=0x7fa3640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d90750357a4fa44 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa363e00640 1 -- 172.21.15.2:0/3092472439 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa3580ef3d0 con 0x7fa36c1086b0 2024-03-11T18:38:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa36b400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 0x7fa3500a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa360c00640 1 -- 172.21.15.2:0/3590735708 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004d20 con 0x557bff59eb20 2024-03-11T18:38:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa360c00640 1 -- 172.21.15.2:0/3590735708 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa364004e80 con 0x557bff59eb20 2024-03-11T18:38:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.592+0000 7fa360c00640 1 -- 172.21.15.2:0/3590735708 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364005280 con 0x557bff59eb20 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 -- 172.21.15.2:0/3590735708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 msgr2=0x557bff59d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa36b400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 0x7fa3500a40d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x557bff59f280 tx=0x7fa35c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa364009f50 tx=0x7fa3640118d0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 -- 172.21.15.2:0/3590735708 shutdown_connections 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff5a7330 0x557bff5a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59ddb0 0x557bff59e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.593+0000 7fa375235740 1 --2- 172.21.15.2:0/3590735708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff59d870 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.594+0000 7fa375235740 1 -- 172.21.15.2:0/3590735708 >> 172.21.15.2:0/3590735708 conn(0x557bff5654b0 msgr2=0x557bff598aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.594+0000 7fa375235740 1 -- 172.21.15.2:0/3590735708 shutdown_connections 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.594+0000 7fa375235740 1 -- 172.21.15.2:0/3590735708 wait complete. 2024-03-11T18:38:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.594+0000 7fa375235740 1 -- start start 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.594+0000 7fa375235740 1 Processor -- start 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5a4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff5045d0 con 0x557bff59ddb0 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff4a9b50 con 0x557bff5a7330 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff4a97f0 con 0x557bff59eb20 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa375235740 10 client.0 _ll_register_callbacks cb 0x557bff4bd550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa36b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa36b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53084/0 (socket says 172.21.15.2:53084) 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa36b400640 1 -- 172.21.15.2:0/3254958524 learned_addr learned my addr 172.21.15.2:0/3254958524 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa36be00640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.595+0000 7fa36aa00640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 -- 172.21.15.2:0/3254958524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 msgr2=0x557bff5a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 -- 172.21.15.2:0/3254958524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 msgr2=0x557bff5a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 -- 172.21.15.2:0/3254958524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa364009c00 con 0x557bff5a7330 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa36be00640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa364011dd0 tx=0x7fa364004cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ceba1078f8f1af server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa355600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.596+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004850 con 0x557bff5a7330 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa364005000 con 0x557bff5a7330 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557bff4d6cd0 con 0x557bff5a7330 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3640051c0 con 0x557bff5a7330 2024-03-11T18:38:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa375235740 10 client.25871 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557bff502c50 con 0x557bff5a7330 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557bff4d6cd0 con 0x557bff5a7330 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa36402b020 con 0x557bff5a7330 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa375235740 10 client.25871 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.597+0000 7fa375235740 10 client.25871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa375235740 20 client.25871 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff5b72a0 0x557bff5b9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa375235740 4 client.25871 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bff4face0 con 0x557bff5b72a0 2024-03-11T18:38:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa36b400640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff5b72a0 0x557bff5b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa36b400640 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff5b72a0 0x557bff5b9690 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad7c31ef33767b3a server_cookie=b2e001c755ac5ed7 in_seq=0 out_seq=0 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.598+0000 7fa355600640 10 client.25871 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa355600640 1 -- 172.21.15.2:0/3254958524 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557bff4face0 con 0x557bff5b72a0 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa355600640 10 client.25871 handle_command_reply: tid=0 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 client.25871 shutdown 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 msgr2=0x557bff5a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa364011dd0 tx=0x7fa364004cf0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 shutdown_connections 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff5b72a0 0x557bff5b9690 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5a4dd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa355600640 0 client.25871 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 --2- 172.21.15.2:0/3254958524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 >> 172.21.15.2:0/3254958524 conn(0x557bff5654b0 msgr2=0x557bff59a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 shutdown_connections 2024-03-11T18:38:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 1 -- 172.21.15.2:0/3254958524 wait complete. 2024-03-11T18:38:40.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.600+0000 7fa375235740 20 client.25871 trim_cache size 0 max 16384 2024-03-11T18:38:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.615+0000 7fa375235740 1 Processor -- start 2024-03-11T18:38:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.615+0000 7fa375235740 1 -- start start 2024-03-11T18:38:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.627 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 0x557bff5a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff24ce60 con 0x557bff59ddb0 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff4face0 con 0x557bff59eb20 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa375235740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff4d6cd0 con 0x557bff5a7330 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37132/0 (socket says 172.21.15.2:37132) 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 -- 172.21.15.2:0/2855004476 learned_addr learned my addr 172.21.15.2:0/2855004476 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 0x557bff5a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36b400640 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 -- 172.21.15.2:0/2855004476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 msgr2=0x557bff5a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 0x557bff5a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 -- 172.21.15.2:0/2855004476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 msgr2=0x557bff5a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36aa00640 1 -- 172.21.15.2:0/2855004476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bff49f530 con 0x557bff59eb20 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.616+0000 7fa36be00640 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 0x557bff5a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa36aa00640 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa358049040 tx=0x7fa3580bbb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a9364b49b251ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa357e00640 1 -- 172.21.15.2:0/2855004476 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3580bbdc0 con 0x557bff59eb20 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa357e00640 1 -- 172.21.15.2:0/2855004476 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3580bfe60 con 0x557bff59eb20 2024-03-11T18:38:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa357e00640 1 -- 172.21.15.2:0/2855004476 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35802c760 con 0x557bff59eb20 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa375235740 1 -- 172.21.15.2:0/2855004476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 msgr2=0x557bff5a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.617+0000 7fa375235740 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa358049040 tx=0x7fa3580bbb90 comp rx=0 tx=0).stop 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 -- 172.21.15.2:0/2855004476 shutdown_connections 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff5a7330 0x557bff5a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff59eb20 0x557bff5a4de0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 --2- 172.21.15.2:0/2855004476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 -- 172.21.15.2:0/2855004476 >> 172.21.15.2:0/2855004476 conn(0x557bff5654b0 msgr2=0x557bff5a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 -- 172.21.15.2:0/2855004476 shutdown_connections 2024-03-11T18:38:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 -- 172.21.15.2:0/2855004476 wait complete. 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 -- start start 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.618+0000 7fa375235740 1 Processor -- start 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff5ad4c0 con 0x557bff59ddb0 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff5ad630 con 0x557bff5a7330 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bff615f80 con 0x557bff59eb20 2024-03-11T18:38:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa375235740 10 client.0 _ll_register_callbacks cb 0x7fa35c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37144/0 (socket says 172.21.15.2:37144) 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 -- 172.21.15.2:0/1432381802 learned_addr learned my addr 172.21.15.2:0/1432381802 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36aa00640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 -- 172.21.15.2:0/1432381802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 msgr2=0x557bff5ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 -- 172.21.15.2:0/1432381802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 msgr2=0x557bff5ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.619+0000 7fa36be00640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa36be00640 1 -- 172.21.15.2:0/1432381802 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bff502c50 con 0x557bff5a7330 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa36aa00640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa36be00640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa364004020 tx=0x7fa36402eae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a263d54b0703bc71 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa355600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364004360 con 0x557bff5a7330 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa364003c40 con 0x557bff5a7330 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557bff529d30 con 0x557bff5a7330 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3640044c0 con 0x557bff5a7330 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa375235740 10 client.25877 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.620+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fa36c1a6850 con 0x557bff5a7330 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557bff529d30 con 0x557bff5a7330 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa36402b040 con 0x557bff5a7330 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 10 client.25877 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 10 client.25877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 20 client.25877 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff617220 0x557bff619640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 4 client.25877 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557bff204600 con 0x557bff617220 2024-03-11T18:38:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.621+0000 7fa36b400640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff617220 0x557bff619640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.622+0000 7fa36b400640 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff617220 0x557bff619640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=158f8223ab14622d server_cookie=3c495cfcf7cb1945 in_seq=0 out_seq=0 2024-03-11T18:38:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.622+0000 7fa355600640 10 client.25877 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.845+0000 7fa355600640 1 -- 172.21.15.2:0/1432381802 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557bff204600 con 0x557bff617220 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.845+0000 7fa355600640 10 client.25877 handle_command_reply: tid=0 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.845+0000 7fa375235740 1 client.25877 shutdown 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.845+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 msgr2=0x557bff5acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.845+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa364004020 tx=0x7fa36402eae0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 shutdown_connections 2024-03-11T18:38:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557bff617220 0x557bff619640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557bff5a7330 0x557bff5acdc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557bff59eb20 0x557bff5ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 --2- 172.21.15.2:0/1432381802 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557bff59ddb0 0x557bff5ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 >> 172.21.15.2:0/1432381802 conn(0x557bff5654b0 msgr2=0x557bff597d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa355600640 0 client.25877 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 shutdown_connections 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 1 -- 172.21.15.2:0/1432381802 wait complete. 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.846+0000 7fa375235740 20 client.25877 trim_cache size 0 max 16384 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:38:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 msgr2=0x7fa3500a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 0x7fa3500a40d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x557bff59f280 tx=0x7fa35c005f90 comp rx=0 tx=0).stop 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 msgr2=0x7fa36c19fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa358000c00 tx=0x7fa358004510 comp rx=0 tx=0).stop 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 shutdown_connections 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa36c199850 0x7fa36c199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa36c1086b0 0x7fa36c19fc20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa3500a1c10 0x7fa3500a40d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 --2- 172.21.15.2:0/3092472439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa36c1035e0 0x7fa36c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 >> 172.21.15.2:0/3092472439 conn(0x7fa36c0fee00 msgr2=0x7fa36c100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 shutdown_connections 2024-03-11T18:38:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:40.847+0000 7fa372400640 1 -- 172.21.15.2:0/3092472439 wait complete. 2024-03-11T18:38:41.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 366149 8.53 MB/sec execute 101 sec latency 120.321 ms 2024-03-11T18:38:42.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:41 smithi002 ceph-mon[31328]: pgmap v152: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:42.403 INFO:tasks.workunit.client.0.smithi002.stdout: 1 368106 8.55 MB/sec execute 102 sec latency 67.866 ms 2024-03-11T18:38:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:41 smithi165 ceph-mon[36634]: pgmap v152: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:42.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:41 smithi023 ceph-mon[36406]: pgmap v152: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:43.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 369560 8.54 MB/sec execute 103 sec latency 13.067 ms 2024-03-11T18:38:44.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:43 smithi002 ceph-mon[31328]: pgmap v153: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:44.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 371274 8.53 MB/sec execute 104 sec latency 17.243 ms 2024-03-11T18:38:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:43 smithi165 ceph-mon[36634]: pgmap v153: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:44.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:43 smithi023 ceph-mon[36406]: pgmap v153: 129 pgs: 129 active+clean; 60 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-03-11T18:38:45.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 372839 8.56 MB/sec execute 105 sec latency 18.293 ms 2024-03-11T18:38:46.207 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:45 smithi165 ceph-mon[36634]: pgmap v154: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-03-11T18:38:46.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:45 smithi002 ceph-mon[31328]: pgmap v154: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-03-11T18:38:46.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 374126 8.53 MB/sec execute 106 sec latency 15.960 ms 2024-03-11T18:38:46.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:45 smithi023 ceph-mon[36406]: pgmap v154: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-03-11T18:38:47.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 376121 8.57 MB/sec execute 107 sec latency 31.845 ms 2024-03-11T18:38:48.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:47 smithi002 ceph-mon[31328]: pgmap v155: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 279 op/s 2024-03-11T18:38:48.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 376964 8.51 MB/sec execute 108 sec latency 18.325 ms 2024-03-11T18:38:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:47 smithi165 ceph-mon[36634]: pgmap v155: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 279 op/s 2024-03-11T18:38:48.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:47 smithi023 ceph-mon[36406]: pgmap v155: 129 pgs: 129 active+clean; 56 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 2.0 MiB/s wr, 279 op/s 2024-03-11T18:38:49.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 379035 8.54 MB/sec execute 109 sec latency 13.748 ms 2024-03-11T18:38:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:49 smithi002 ceph-mon[31328]: pgmap v156: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 363 op/s 2024-03-11T18:38:50.404 INFO:tasks.workunit.client.0.smithi002.stdout: 1 380302 8.52 MB/sec execute 110 sec latency 18.365 ms 2024-03-11T18:38:50.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:49 smithi165 ceph-mon[36634]: pgmap v156: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 363 op/s 2024-03-11T18:38:50.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:49 smithi023 ceph-mon[36406]: pgmap v156: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 363 op/s 2024-03-11T18:38:51.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 382054 8.51 MB/sec execute 111 sec latency 22.925 ms 2024-03-11T18:38:52.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:51 smithi002 ceph-mon[31328]: pgmap v157: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:52.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 383480 8.54 MB/sec execute 112 sec latency 67.247 ms 2024-03-11T18:38:52.405 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:51 smithi023 ceph-mon[36406]: pgmap v157: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:51 smithi165 ceph-mon[36634]: pgmap v157: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:53.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 384529 8.51 MB/sec execute 113 sec latency 18.672 ms 2024-03-11T18:38:54.010 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 -- 172.21.15.2:0/2614649889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 msgr2=0x7f3e201027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 --2- 172.21.15.2:0/2614649889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e201027d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009ca0 tx=0x7f3e1802c190 comp rx=0 tx=0).stop 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 -- 172.21.15.2:0/2614649889 shutdown_connections 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 --2- 172.21.15.2:0/2614649889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 0x7f3e2010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 --2- 172.21.15.2:0/2614649889 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 0x7f3e2010b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 --2- 172.21.15.2:0/2614649889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e201027d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.008+0000 7f3e26200640 1 -- 172.21.15.2:0/2614649889 >> 172.21.15.2:0/2614649889 conn(0x7f3e20077190 msgr2=0x7f3e200775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.009+0000 7f3e26200640 1 -- 172.21.15.2:0/2614649889 shutdown_connections 2024-03-11T18:38:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.009+0000 7f3e26200640 1 -- 172.21.15.2:0/2614649889 wait complete. 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 Processor -- start 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 -- start start 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 0x7f3e2019d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 0x7f3e201975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20197c40 con 0x7f3e2010b7c0 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20197db0 con 0x7f3e201023f0 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e26200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20197f70 con 0x7f3e20102d10 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58916/0 (socket says 172.21.15.2:58916) 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e1f400640 1 -- 172.21.15.2:0/2778505010 learned_addr learned my addr 172.21.15.2:0/2778505010 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 0x7f3e201975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.010+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 0x7f3e2019d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 -- 172.21.15.2:0/2778505010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 msgr2=0x7f3e2019d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 0x7f3e2019d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 -- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 msgr2=0x7f3e201975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 0x7f3e201975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 -- 172.21.15.2:0/2778505010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e18009900 con 0x7f3e201023f0 2024-03-11T18:38:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e18005e30 tx=0x7f3e180346d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.011+0000 7f3e17e00640 1 -- 172.21.15.2:0/2778505010 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18003d80 con 0x7f3e201023f0 2024-03-11T18:38:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.012+0000 7f3e17e00640 1 -- 172.21.15.2:0/2778505010 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e1802cdb0 con 0x7f3e201023f0 2024-03-11T18:38:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.012+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e201981f0 con 0x7f3e201023f0 2024-03-11T18:38:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.012+0000 7f3e17e00640 1 -- 172.21.15.2:0/2778505010 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18004580 con 0x7f3e201023f0 2024-03-11T18:38:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.012+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e201a44e0 con 0x7f3e201023f0 2024-03-11T18:38:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.016+0000 7f3e17e00640 1 -- 172.21.15.2:0/2778505010 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f3e1803b050 con 0x7f3e201023f0 2024-03-11T18:38:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.016+0000 7f3e28fab740 1 Processor -- start 2024-03-11T18:38:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.016+0000 7f3e28fab740 1 -- start start 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19522b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 0x55a195237720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1951e7620 con 0x55a195235330 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1951998c0 con 0x55a19522bdb0 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e28fab740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a195196df0 con 0x55a19522cb20 2024-03-11T18:38:54.020 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58924/0 (socket says 172.21.15.2:58924) 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 -- 172.21.15.2:0/2332411115 learned_addr learned my addr 172.21.15.2:0/2332411115 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 0x55a195237720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19522b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 -- 172.21.15.2:0/2332411115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 msgr2=0x55a19522b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19522b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 -- 172.21.15.2:0/2332411115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 msgr2=0x55a195237720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 0x55a195237720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1f400640 1 -- 172.21.15.2:0/2332411115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a195113b00 con 0x55a19522bdb0 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19522b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.017+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 0x55a195237720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e17e00640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 0x7f3e040a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e1f400640 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ecf70 tx=0x7f3e180f3c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6992aaa211b7ed5d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e17e00640 1 -- 172.21.15.2:0/2778505010 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3e180efbd0 con 0x7f3e201023f0 2024-03-11T18:38:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 0x7f3e040a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e0be00640 1 -- 172.21.15.2:0/2332411115 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180b8030 con 0x55a19522bdb0 2024-03-11T18:38:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e0be00640 1 -- 172.21.15.2:0/2332411115 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e180f3e80 con 0x55a19522bdb0 2024-03-11T18:38:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.018+0000 7f3e0be00640 1 -- 172.21.15.2:0/2332411115 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180bc390 con 0x55a19522bdb0 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 -- 172.21.15.2:0/2332411115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 msgr2=0x55a19522c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 0x7f3e040a4050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000b710 tx=0x7f3e1000b2e0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ecf70 tx=0x7f3e180f3c30 comp rx=0 tx=0).stop 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 -- 172.21.15.2:0/2332411115 shutdown_connections 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a195235330 0x55a195237720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a19522c230 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 --2- 172.21.15.2:0/2332411115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19522b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 -- 172.21.15.2:0/2332411115 >> 172.21.15.2:0/2332411115 conn(0x55a1951f34b0 msgr2=0x55a195226aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 -- 172.21.15.2:0/2332411115 shutdown_connections 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.019+0000 7f3e28fab740 1 -- 172.21.15.2:0/2332411115 wait complete. 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.020+0000 7f3e28fab740 1 -- start start 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.020+0000 7f3e28fab740 1 Processor -- start 2024-03-11T18:38:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195239320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1951925d0 con 0x55a19522bdb0 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a195137b50 con 0x55a19522cb20 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1951377f0 con 0x55a195235330 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e28fab740 10 client.0 _ll_register_callbacks cb 0x55a19514b550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58926/0 (socket says 172.21.15.2:58926) 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1ea00640 1 -- 172.21.15.2:0/1517781030 learned_addr learned my addr 172.21.15.2:0/1517781030 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195239320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 -- 172.21.15.2:0/1517781030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 msgr2=0x55a195239320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195239320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 -- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 msgr2=0x55a19523f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1f400640 1 -- 172.21.15.2:0/1517781030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e201a44e0 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.021+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.022+0000 7f3e1f400640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523ef10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e180f3930 tx=0x7f3e18009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34de222973b1d606 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.022+0000 7f3e08c00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.022+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180b8070 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.022+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3e180bc5c0 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.022+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a195164cd0 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e180f31b0 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 10 client.15762 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a195190c50 con 0x55a19522bdb0 2024-03-11T18:38:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a195164cd0 con 0x55a19522bdb0 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3e180b8c30 con 0x55a19522bdb0 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 10 client.15762 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 20 client.15762 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a195248fe0 0x55a19524b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 4 client.15762 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.023+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a195188ce0 con 0x55a195248fe0 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.024+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a195248fe0 0x55a19524b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.024+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a195248fe0 0x55a19524b3d0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cddb8b88d9ed236 server_cookie=bca0358481b4d2e in_seq=0 out_seq=0 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.024+0000 7f3e08c00640 10 client.15762 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.025+0000 7f3e08c00640 1 -- 172.21.15.2:0/1517781030 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a195188ce0 con 0x55a195248fe0 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.025+0000 7f3e08c00640 10 client.15762 handle_command_reply: tid=0 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.025+0000 7f3e28fab740 1 client.15762 shutdown 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 msgr2=0x55a19523ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523ef10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e180f3930 tx=0x7f3e18009800 comp rx=0 tx=0).stop 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 shutdown_connections 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a195248fe0 0x55a19524b3d0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195239320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522cb20 0x55a19523f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 --2- 172.21.15.2:0/1517781030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523ef10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 >> 172.21.15.2:0/1517781030 conn(0x55a1951f34b0 msgr2=0x55a1952287f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e08c00640 0 client.15762 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 shutdown_connections 2024-03-11T18:38:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 1 -- 172.21.15.2:0/1517781030 wait complete. 2024-03-11T18:38:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.026+0000 7f3e28fab740 20 client.15762 trim_cache size 0 max 16384 2024-03-11T18:38:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 Processor -- start 2024-03-11T18:38:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 -- start start 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 0x55a195237dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195238300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a194edae60 con 0x55a19522cb20 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1952316e0 con 0x55a19522bdb0 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e28fab740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a19522b080 con 0x55a195235330 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e1f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58940/0 (socket says 172.21.15.2:58940) 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e1f400640 1 -- 172.21.15.2:0/3518879963 learned_addr learned my addr 172.21.15.2:0/3518879963 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.044+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 0x55a195237dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195238300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 -- 172.21.15.2:0/3518879963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 msgr2=0x55a195238300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195238300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 -- 172.21.15.2:0/3518879963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 msgr2=0x55a195237dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 0x55a195237dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 -- 172.21.15.2:0/3518879963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a19512d530 con 0x55a19522bdb0 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 0x55a195237dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:38:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e1f400640 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ed090 tx=0x7f3e180baad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75354e70a88dfe7d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e0b400640 1 -- 172.21.15.2:0/3518879963 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180b8040 con 0x55a19522bdb0 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e0b400640 1 -- 172.21.15.2:0/3518879963 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e180efe10 con 0x55a19522bdb0 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.045+0000 7f3e0b400640 1 -- 172.21.15.2:0/3518879963 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180a7070 con 0x55a19522bdb0 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 -- 172.21.15.2:0/3518879963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 msgr2=0x55a1952392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ed090 tx=0x7f3e180baad0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 -- 172.21.15.2:0/3518879963 shutdown_connections 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a195235330 0x55a195238300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522cb20 0x55a195237dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 --2- 172.21.15.2:0/3518879963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a19522bdb0 0x55a1952392c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 -- 172.21.15.2:0/3518879963 >> 172.21.15.2:0/3518879963 conn(0x55a1951f34b0 msgr2=0x55a195238bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 -- 172.21.15.2:0/3518879963 shutdown_connections 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.046+0000 7f3e28fab740 1 -- 172.21.15.2:0/3518879963 wait complete. 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.047+0000 7f3e28fab740 1 -- start start 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.047+0000 7f3e28fab740 1 Processor -- start 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.047+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19523d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a19523d8a0 con 0x55a19522bdb0 2024-03-11T18:38:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a19523da10 con 0x55a195235330 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a19529e980 con 0x55a19522cb20 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e28fab740 10 client.0 _ll_register_callbacks cb 0x7f3e180f39f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58956/0 (socket says 172.21.15.2:58956) 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 -- 172.21.15.2:0/881345219 learned_addr learned my addr 172.21.15.2:0/881345219 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1f400640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1ea00640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19523d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 -- 172.21.15.2:0/881345219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 msgr2=0x55a19523d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19523d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 -- 172.21.15.2:0/881345219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 msgr2=0x55a19523cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.048+0000 7f3e1fe00640 1 -- 172.21.15.2:0/881345219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c012650 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e1fe00640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e10003fb0 tx=0x7f3e10014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f2384fe4d5360b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e08c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e10022040 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f3e10005220 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a1951b7d30 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e100053e0 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.049+0000 7f3e28fab740 10 client.25895 fetch_fsmap learned FSMap version 27 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a195190c50 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a1951b7d30 con 0x55a195235330 2024-03-11T18:38:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3e10025070 con 0x55a195235330 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 10 client.25895 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 10 client.25895 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 20 client.25895 populate_metadata read hostname 'smithi002' 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a1952a0200 0x55a1952a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 4 client.25895 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a194e92600 con 0x55a1952a0200 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.050+0000 7f3e1f400640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a1952a0200 0x55a1952a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.051+0000 7f3e1f400640 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a1952a0200 0x55a1952a2620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbab3f7f0101b91d server_cookie=51325c713dde2732 in_seq=0 out_seq=0 2024-03-11T18:38:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.051+0000 7f3e08c00640 10 client.25895 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:54.208 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:53 smithi023 ceph-mon[36406]: pgmap v158: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:54.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:53 smithi002 ceph-mon[31328]: pgmap v158: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e08c00640 1 -- 172.21.15.2:0/881345219 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a194e92600 con 0x55a1952a0200 2024-03-11T18:38:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e08c00640 10 client.25895 handle_command_reply: tid=0 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 client.25895 shutdown 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 msgr2=0x55a19529e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e10003fb0 tx=0x7f3e10014040 comp rx=0 tx=0).stop 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 shutdown_connections 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a1952a0200 0x55a1952a2620 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a195235330 0x55a19529e440 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a19522cb20 0x55a19523d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 --2- 172.21.15.2:0/881345219 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a19522bdb0 0x55a19523cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 >> 172.21.15.2:0/881345219 conn(0x55a1951f34b0 msgr2=0x55a195238bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e08c00640 0 client.25895 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 shutdown_connections 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.266+0000 7f3e28fab740 1 -- 172.21.15.2:0/881345219 wait complete. 2024-03-11T18:38:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e28fab740 20 client.25895 trim_cache size 0 max 16384 2024-03-11T18:38:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:38:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:38:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:38:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 msgr2=0x7f3e040a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 0x7f3e040a4050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000b710 tx=0x7f3e1000b2e0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 msgr2=0x7f3e2019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e18005e30 tx=0x7f3e180346d0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.267+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 shutdown_connections 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f3e040a1b90 0x7f3e040a4050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3e2010b7c0 0x7f3e201975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3e20102d10 0x7f3e2019d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 --2- 172.21.15.2:0/2778505010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e201023f0 0x7f3e2019d440 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 >> 172.21.15.2:0/2778505010 conn(0x7f3e20077190 msgr2=0x7f3e200fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 shutdown_connections 2024-03-11T18:38:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:38:54.268+0000 7f3e26200640 1 -- 172.21.15.2:0/2778505010 wait complete. 2024-03-11T18:38:54.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 386492 8.52 MB/sec execute 114 sec latency 163.465 ms 2024-03-11T18:38:54.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:53 smithi165 ceph-mon[36634]: pgmap v158: 129 pgs: 129 active+clean; 61 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-03-11T18:38:55.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 387468 8.49 MB/sec execute 115 sec latency 21.650 ms 2024-03-11T18:38:56.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:55 smithi002 ceph-mon[31328]: pgmap v159: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 368 op/s 2024-03-11T18:38:56.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:55 smithi165 ceph-mon[36634]: pgmap v159: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 368 op/s 2024-03-11T18:38:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:55 smithi023 ceph-mon[36406]: pgmap v159: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 368 op/s 2024-03-11T18:38:56.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 389553 8.51 MB/sec execute 116 sec latency 20.819 ms 2024-03-11T18:38:57.405 INFO:tasks.workunit.client.0.smithi002.stdout: 1 390772 8.50 MB/sec execute 117 sec latency 77.565 ms 2024-03-11T18:38:58.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:57 smithi002 ceph-mon[31328]: pgmap v160: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-11T18:38:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:57 smithi165 ceph-mon[36634]: pgmap v160: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-11T18:38:58.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 392588 8.49 MB/sec execute 118 sec latency 18.342 ms 2024-03-11T18:38:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:57 smithi023 ceph-mon[36406]: pgmap v160: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 1.9 MiB/s wr, 169 op/s 2024-03-11T18:38:59.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:58 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:38:59.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:58 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:38:59.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 394170 8.52 MB/sec execute 119 sec latency 10.591 ms 2024-03-11T18:38:59.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:58 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:39:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:59 smithi002 ceph-mon[31328]: pgmap v161: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 365 op/s 2024-03-11T18:39:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:59 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:39:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:59 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:39:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:38:59 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:39:00.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:59 smithi165 ceph-mon[36634]: pgmap v161: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 365 op/s 2024-03-11T18:39:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:59 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:39:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:59 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:39:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:38:59 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:39:00.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 395628 8.49 MB/sec execute 120 sec latency 33.775 ms 2024-03-11T18:39:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:59 smithi023 ceph-mon[36406]: pgmap v161: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 365 op/s 2024-03-11T18:39:00.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:59 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:39:00.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:59 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:39:00.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:38:59 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:39:01.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 397482 8.53 MB/sec execute 121 sec latency 65.018 ms 2024-03-11T18:39:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:01 smithi002 ceph-mon[31328]: pgmap v162: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:02.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:01 smithi165 ceph-mon[36634]: pgmap v162: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:02.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 397508 8.46 MB/sec execute 122 sec latency 149.091 ms 2024-03-11T18:39:02.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:01 smithi023 ceph-mon[36406]: pgmap v162: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:03.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 398368 8.41 MB/sec execute 123 sec latency 84.529 ms 2024-03-11T18:39:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:03 smithi002 ceph-mon[31328]: pgmap v163: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:04.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:03 smithi165 ceph-mon[36634]: pgmap v163: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:04.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:03 smithi023 ceph-mon[36406]: pgmap v163: 129 pgs: 129 active+clean; 59 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.1 MiB/s wr, 281 op/s 2024-03-11T18:39:04.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 400816 8.47 MB/sec execute 124 sec latency 15.345 ms 2024-03-11T18:39:05.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 401771 8.42 MB/sec execute 125 sec latency 67.329 ms 2024-03-11T18:39:06.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:05 smithi002 ceph-mon[31328]: pgmap v164: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-11T18:39:06.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:05 smithi165 ceph-mon[36634]: pgmap v164: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-11T18:39:06.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:05 smithi023 ceph-mon[36406]: pgmap v164: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 390 op/s 2024-03-11T18:39:06.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 404208 8.46 MB/sec execute 126 sec latency 13.559 ms 2024-03-11T18:39:07.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 405142 8.44 MB/sec execute 127 sec latency 85.814 ms 2024-03-11T18:39:07.421 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 -- 172.21.15.2:0/1175040329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c1013a0 msgr2=0x7fe13c101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 --2- 172.21.15.2:0/1175040329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c1013a0 0x7fe13c101780 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009930 tx=0x7fe12c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 -- 172.21.15.2:0/1175040329 shutdown_connections 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 --2- 172.21.15.2:0/1175040329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c106470 0x7fe13c10e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 --2- 172.21.15.2:0/1175040329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c101d50 0x7fe13c105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 --2- 172.21.15.2:0/1175040329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c1013a0 0x7fe13c101780 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.419+0000 7fe142400640 1 -- 172.21.15.2:0/1175040329 >> 172.21.15.2:0/1175040329 conn(0x7fe13c0fcc00 msgr2=0x7fe13c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.420+0000 7fe142400640 1 -- 172.21.15.2:0/1175040329 shutdown_connections 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.420+0000 7fe142400640 1 -- 172.21.15.2:0/1175040329 wait complete. 2024-03-11T18:39:07.422 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.421+0000 7fe142400640 1 Processor -- start 2024-03-11T18:39:07.423 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.421+0000 7fe142400640 1 -- start start 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.421+0000 7fe142400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c1013a0 0x7fe13c073480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.421+0000 7fe142400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.421+0000 7fe142400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 0x7fe13c073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe142400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c1aa3f0 con 0x7fe13c1013a0 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe142400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c1aa560 con 0x7fe13c106470 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe142400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13c1aa720 con 0x7fe13c075120 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:33156/0 (socket says 172.21.15.2:33156) 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13aa00640 1 -- 172.21.15.2:0/2355715676 learned_addr learned my addr 172.21.15.2:0/2355715676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 0x7fe13c073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.424 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13b400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c1013a0 0x7fe13c073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.425 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 -- 172.21.15.2:0/2355715676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 msgr2=0x7fe13c0739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 -- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c1013a0 msgr2=0x7fe13c073480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c1013a0 0x7fe13c073480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13be00640 1 -- 172.21.15.2:0/2355715676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c009560 con 0x7fe13c075120 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.422+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 0x7fe13c073f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe134009fd0 tx=0x7fe13400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe133e00640 1 -- 172.21.15.2:0/2355715676 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe13400ce20 con 0x7fe13c075120 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe133e00640 1 -- 172.21.15.2:0/2355715676 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe134005020 con 0x7fe13c075120 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13c1aaa00 con 0x7fe13c075120 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe133e00640 1 -- 172.21.15.2:0/2355715676 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134010690 con 0x7fe13c075120 2024-03-11T18:39:07.426 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.423+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13c1aaf60 con 0x7fe13c075120 2024-03-11T18:39:07.428 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.426+0000 7fe133e00640 1 -- 172.21.15.2:0/2355715676 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fe134005500 con 0x7fe13c075120 2024-03-11T18:39:07.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.427+0000 7fe145165740 1 Processor -- start 2024-03-11T18:39:07.429 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.427+0000 7fe145165740 1 -- start start 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac44870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac50720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15ac00620 con 0x55b15ac4e330 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15abb28c0 con 0x55b15ac44db0 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe145165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15abafdf0 con 0x55b15ac45b20 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:33158/0 (socket says 172.21.15.2:33158) 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 -- 172.21.15.2:0/1189036885 learned_addr learned my addr 172.21.15.2:0/1189036885 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:07.431 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe133e00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 0x7fe1200a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13aa00640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac44870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13aa00640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac50720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 -- 172.21.15.2:0/1189036885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 msgr2=0x55b15ac44870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe133e00640 1 -- 172.21.15.2:0/2355715676 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe134014070 con 0x7fe13c075120 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac44870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 -- 172.21.15.2:0/1189036885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 msgr2=0x55b15ac50720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac50720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13b400640 1 -- 172.21.15.2:0/1189036885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b15ab2cb00 con 0x55b15ac44db0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13aa00640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac50720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.428+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 0x7fe1200a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.429+0000 7fe13b400640 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005ab0 tx=0x7fe12c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb0896fef5c2efc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.429+0000 7fe130c00640 1 -- 172.21.15.2:0/1189036885 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c0048b0 con 0x55b15ac44db0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.429+0000 7fe130c00640 1 -- 172.21.15.2:0/1189036885 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe12c004a10 con 0x55b15ac44db0 2024-03-11T18:39:07.432 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.429+0000 7fe130c00640 1 -- 172.21.15.2:0/1189036885 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c005170 con 0x55b15ac44db0 2024-03-11T18:39:07.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 -- 172.21.15.2:0/1189036885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 msgr2=0x55b15ac45230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005ab0 tx=0x7fe12c0046c0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe13be00640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 0x7fe1200a40d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c920 tx=0x7fe13400c890 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 -- 172.21.15.2:0/1189036885 shutdown_connections 2024-03-11T18:39:07.435 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac50720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac45230 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 --2- 172.21.15.2:0/1189036885 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac44870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 -- 172.21.15.2:0/1189036885 >> 172.21.15.2:0/1189036885 conn(0x55b15ac0c4b0 msgr2=0x55b15ac3faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 -- 172.21.15.2:0/1189036885 shutdown_connections 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.430+0000 7fe145165740 1 -- 172.21.15.2:0/1189036885 wait complete. 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 -- start start 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 Processor -- start 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac4b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac4be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15abab5d0 con 0x55b15ac4e330 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15ab50b50 con 0x55b15ac44db0 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15ab507f0 con 0x55b15ac45b20 2024-03-11T18:39:07.436 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.431+0000 7fe145165740 10 client.0 _ll_register_callbacks cb 0x55b15ab64550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac4b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:54360/0 (socket says 172.21.15.2:54360) 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 -- 172.21.15.2:0/2321414858 learned_addr learned my addr 172.21.15.2:0/2321414858 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac4be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 -- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 msgr2=0x55b15ac4b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac4b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 -- 172.21.15.2:0/2321414858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 msgr2=0x55b15ac4be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac4be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 -- 172.21.15.2:0/2321414858 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13c10e850 con 0x55b15ac45b20 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe13b400640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005ec0 tx=0x7fe12c02cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b67c616365ccd9a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.432+0000 7fe129600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c02cca0 con 0x55b15ac45b20 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe12c005300 con 0x55b15ac45b20 2024-03-11T18:39:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b15ab7dcd0 con 0x55b15ac45b20 2024-03-11T18:39:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12c0054c0 con 0x55b15ac45b20 2024-03-11T18:39:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe145165740 10 client.25903 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b15aba9c50 con 0x55b15ac45b20 2024-03-11T18:39:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.433+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b15ab7dcd0 con 0x55b15ac45b20 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe12c03c050 con 0x55b15ac45b20 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 10 client.25903 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 10 client.25903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 20 client.25903 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15ac5dfb0 0x55b15ac603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 4 client.25903 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b15aba1ce0 con 0x55b15ac5dfb0 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15ac5dfb0 0x55b15ac603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15ac5dfb0 0x55b15ac603d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7ad8770c47a7134 server_cookie=8db8c0df821c7c05 in_seq=0 out_seq=0 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.434+0000 7fe129600640 10 client.25903 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe129600640 1 -- 172.21.15.2:0/2321414858 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b15aba1ce0 con 0x55b15ac5dfb0 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe129600640 10 client.25903 handle_command_reply: tid=0 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe145165740 1 client.25903 shutdown 2024-03-11T18:39:07.440 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 msgr2=0x55b15ac4b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005ec0 tx=0x7fe12c02cb30 comp rx=0 tx=0).stop 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 shutdown_connections 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.435+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15ac5dfb0 0x55b15ac603d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15ac4be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15ac4b8c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 --2- 172.21.15.2:0/2321414858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15ac4b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe129600640 0 client.25903 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 >> 172.21.15.2:0/2321414858 conn(0x55b15ac0c4b0 msgr2=0x55b15ac417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 shutdown_connections 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 1 -- 172.21.15.2:0/2321414858 wait complete. 2024-03-11T18:39:07.441 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.436+0000 7fe145165740 20 client.25903 trim_cache size 0 max 16384 2024-03-11T18:39:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 Processor -- start 2024-03-11T18:39:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 -- start start 2024-03-11T18:39:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac44db0 0x55b15ac55160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 0x55b15ac485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15a8f3e60 con 0x55b15ac44db0 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15ac44080 con 0x55b15ac45b20 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.451+0000 7fe145165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15ac461c0 con 0x55b15ac4e330 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:33174/0 (socket says 172.21.15.2:33174) 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 -- 172.21.15.2:0/892856627 learned_addr learned my addr 172.21.15.2:0/892856627 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13aa00640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac44db0 0x55b15ac55160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13aa00640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 0x55b15ac485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 -- 172.21.15.2:0/892856627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 msgr2=0x55b15ac485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 0x55b15ac485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 -- 172.21.15.2:0/892856627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac44db0 msgr2=0x55b15ac55160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac44db0 0x55b15ac55160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 -- 172.21.15.2:0/892856627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b15ab46530 con 0x55b15ac45b20 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13aa00640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 0x55b15ac485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.452+0000 7fe13b400640 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe12c004630 tx=0x7fe12c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514ce265d814fd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.458 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.453+0000 7fe12be00640 1 -- 172.21.15.2:0/892856627 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c03f440 con 0x55b15ac45b20 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.453+0000 7fe12be00640 1 -- 172.21.15.2:0/892856627 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe12c03f860 con 0x55b15ac45b20 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.453+0000 7fe12be00640 1 -- 172.21.15.2:0/892856627 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c04a450 con 0x55b15ac45b20 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.453+0000 7fe145165740 1 -- 172.21.15.2:0/892856627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 msgr2=0x55b15ac556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.453+0000 7fe145165740 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe12c004630 tx=0x7fe12c005fd0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 -- 172.21.15.2:0/892856627 shutdown_connections 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac4e330 0x55b15ac485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac45b20 0x55b15ac556a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 --2- 172.21.15.2:0/892856627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac44db0 0x55b15ac55160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 -- 172.21.15.2:0/892856627 >> 172.21.15.2:0/892856627 conn(0x55b15ac0c4b0 msgr2=0x55b15ac46690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.459 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 -- 172.21.15.2:0/892856627 shutdown_connections 2024-03-11T18:39:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 -- 172.21.15.2:0/892856627 wait complete. 2024-03-11T18:39:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 -- start start 2024-03-11T18:39:07.460 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.454+0000 7fe145165740 1 Processor -- start 2024-03-11T18:39:07.463 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15acadda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15aca8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15aca89c0 con 0x55b15ac4e330 2024-03-11T18:39:07.464 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15aca8b30 con 0x55b15ac44db0 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b15aca8cf0 con 0x55b15ac45b20 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe145165740 10 client.0 _ll_register_callbacks cb 0x7fe12c02c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe13aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15acadda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe13b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe13aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15aca8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe13b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:54368/0 (socket says 172.21.15.2:54368) 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.455+0000 7fe13b400640 1 -- 172.21.15.2:0/2209907456 learned_addr learned my addr 172.21.15.2:0/2209907456 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:07.465 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 -- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 msgr2=0x55b15acadda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15acadda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 -- 172.21.15.2:0/2209907456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 msgr2=0x55b15aca8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15aca8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 -- 172.21.15.2:0/2209907456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b15aba9c50 con 0x55b15ac45b20 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe13b400640 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02c240 tx=0x7fe12c005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd1a9d51c8353ede server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe129600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c03f440 con 0x55b15ac45b20 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b15abd0d30 con 0x55b15ac45b20 2024-03-11T18:39:07.466 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.456+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fe12c063020 con 0x55b15ac45b20 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12c0046c0 con 0x55b15ac45b20 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 10 client.25906 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55b15ab6c2c0 con 0x55b15ac45b20 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b15abd0d30 con 0x55b15ac45b20 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe12c03c040 con 0x55b15ac45b20 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 10 client.25906 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 10 client.25906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 20 client.25906 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15acb57a0 0x55b15acb7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 4 client.25906 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:39:07.467 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b15a8ab600 con 0x55b15acb57a0 2024-03-11T18:39:07.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.457+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15acb57a0 0x55b15acb7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:07.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.458+0000 7fe13aa00640 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15acb57a0 0x55b15acb7bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7044504969009b8 server_cookie=308ef2992f772055 in_seq=0 out_seq=0 2024-03-11T18:39:07.468 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.458+0000 7fe129600640 10 client.25906 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.649+0000 7fe129600640 1 -- 172.21.15.2:0/2209907456 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b15a8ab600 con 0x55b15acb57a0 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.649+0000 7fe129600640 10 client.25906 handle_command_reply: tid=0 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.649+0000 7fe145165740 1 client.25906 shutdown 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 msgr2=0x55b15acae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02c240 tx=0x7fe12c005dd0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 shutdown_connections 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b15acb57a0 0x55b15acb7bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b15ac4e330 0x55b15aca8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b15ac45b20 0x55b15acae2e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 --2- 172.21.15.2:0/2209907456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b15ac44db0 0x55b15acadda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 >> 172.21.15.2:0/2209907456 conn(0x55b15ac0c4b0 msgr2=0x55b15ac3e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe129600640 0 client.25906 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.650+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 shutdown_connections 2024-03-11T18:39:07.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.651+0000 7fe145165740 1 -- 172.21.15.2:0/2209907456 wait complete. 2024-03-11T18:39:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.651+0000 7fe145165740 20 client.25906 trim_cache size 0 max 16384 2024-03-11T18:39:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:39:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:39:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:39:07.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.652+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 msgr2=0x7fe1200a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.652+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 0x7fe1200a40d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c920 tx=0x7fe13400c890 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.652+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 msgr2=0x7fe13c073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.652+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 0x7fe13c073f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe134009fd0 tx=0x7fe13400ef10 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 shutdown_connections 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe13c075120 0x7fe13c073f00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe13c106470 0x7fe13c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe1200a1c10 0x7fe1200a40d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 --2- 172.21.15.2:0/2355715676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe13c1013a0 0x7fe13c073480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 >> 172.21.15.2:0/2355715676 conn(0x7fe13c0fcc00 msgr2=0x7fe13c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 shutdown_connections 2024-03-11T18:39:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:07.653+0000 7fe142400640 1 -- 172.21.15.2:0/2355715676 wait complete. 2024-03-11T18:39:08.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:07 smithi002 ceph-mon[31328]: pgmap v165: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 305 op/s 2024-03-11T18:39:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:07 smithi165 ceph-mon[36634]: pgmap v165: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 305 op/s 2024-03-11T18:39:08.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:07 smithi023 ceph-mon[36406]: pgmap v165: 129 pgs: 129 active+clean; 63 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 305 op/s 2024-03-11T18:39:08.406 INFO:tasks.workunit.client.0.smithi002.stdout: 1 406863 8.43 MB/sec execute 128 sec latency 164.640 ms 2024-03-11T18:39:09.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 408380 8.45 MB/sec execute 129 sec latency 13.427 ms 2024-03-11T18:39:10.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:09 smithi002 ceph-mon[31328]: pgmap v166: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 482 op/s 2024-03-11T18:39:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:09 smithi165 ceph-mon[36634]: pgmap v166: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 482 op/s 2024-03-11T18:39:10.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:09 smithi023 ceph-mon[36406]: pgmap v166: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 482 op/s 2024-03-11T18:39:10.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 409590 8.43 MB/sec execute 130 sec latency 16.156 ms 2024-03-11T18:39:11.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 411570 8.46 MB/sec execute 131 sec latency 12.775 ms 2024-03-11T18:39:12.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:12 smithi002 ceph-mon[31328]: pgmap v167: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:12.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:12 smithi165 ceph-mon[36634]: pgmap v167: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:12 smithi023 ceph-mon[36406]: pgmap v167: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:12.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 412299 8.41 MB/sec execute 132 sec latency 44.996 ms 2024-03-11T18:39:13.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 414396 8.43 MB/sec execute 133 sec latency 26.650 ms 2024-03-11T18:39:14.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:14 smithi165 ceph-mon[36634]: pgmap v168: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:14.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:14 smithi023 ceph-mon[36406]: pgmap v168: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:14.407 INFO:tasks.workunit.client.0.smithi002.stdout: 1 415726 8.42 MB/sec execute 134 sec latency 14.807 ms 2024-03-11T18:39:14.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:14 smithi002 ceph-mon[31328]: pgmap v168: 129 pgs: 129 active+clean; 55 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 285 op/s 2024-03-11T18:39:15.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 417519 8.42 MB/sec execute 135 sec latency 15.785 ms 2024-03-11T18:39:16.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:16 smithi023 ceph-mon[36406]: pgmap v169: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 369 op/s 2024-03-11T18:39:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:16 smithi165 ceph-mon[36634]: pgmap v169: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 369 op/s 2024-03-11T18:39:16.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 419119 8.44 MB/sec execute 136 sec latency 10.834 ms 2024-03-11T18:39:16.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:16 smithi002 ceph-mon[31328]: pgmap v169: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 369 op/s 2024-03-11T18:39:17.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 420142 8.41 MB/sec execute 137 sec latency 49.374 ms 2024-03-11T18:39:18.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:18 smithi165 ceph-mon[36634]: pgmap v170: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-03-11T18:39:18.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:18 smithi023 ceph-mon[36406]: pgmap v170: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-03-11T18:39:18.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 422249 8.44 MB/sec execute 138 sec latency 26.543 ms 2024-03-11T18:39:18.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:18 smithi002 ceph-mon[31328]: pgmap v170: 129 pgs: 129 active+clean; 60 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 260 op/s 2024-03-11T18:39:19.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 423055 8.40 MB/sec execute 139 sec latency 17.662 ms 2024-03-11T18:39:20.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:20 smithi165 ceph-mon[36634]: pgmap v171: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 350 op/s 2024-03-11T18:39:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:20 smithi023 ceph-mon[36406]: pgmap v171: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 350 op/s 2024-03-11T18:39:20.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 425046 8.42 MB/sec execute 140 sec latency 16.337 ms 2024-03-11T18:39:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:20 smithi002 ceph-mon[31328]: pgmap v171: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 350 op/s 2024-03-11T18:39:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 -- 172.21.15.2:0/3872777287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c075fd0 msgr2=0x7fa02c111ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 --2- 172.21.15.2:0/3872777287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c075fd0 0x7fa02c111ab0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa018009930 tx=0x7fa01802bc30 comp rx=0 tx=0).stop 2024-03-11T18:39:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 -- 172.21.15.2:0/3872777287 shutdown_connections 2024-03-11T18:39:20.812 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 --2- 172.21.15.2:0/3872777287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c111ff0 0x7fa02c1143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 --2- 172.21.15.2:0/3872777287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c075fd0 0x7fa02c111ab0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 --2- 172.21.15.2:0/3872777287 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c0756b0 0x7fa02c075a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 -- 172.21.15.2:0/3872777287 >> 172.21.15.2:0/3872777287 conn(0x7fa02c0fee00 msgr2=0x7fa02c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 -- 172.21.15.2:0/3872777287 shutdown_connections 2024-03-11T18:39:20.813 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.809+0000 7fa032a00640 1 -- 172.21.15.2:0/3872777287 wait complete. 2024-03-11T18:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.810+0000 7fa032a00640 1 Processor -- start 2024-03-11T18:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.810+0000 7fa032a00640 1 -- start start 2024-03-11T18:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c075fd0 0x7fa02c19e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 0x7fa02c1a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1a2cf0 con 0x7fa02c075fd0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1a2e60 con 0x7fa02c0756b0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa032a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1a3020 con 0x7fa02c111ff0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38322/0 (socket says 172.21.15.2:38322) 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02be00640 1 -- 172.21.15.2:0/3100975865 learned_addr learned my addr 172.21.15.2:0/3100975865 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02b400640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c075fd0 0x7fa02c19e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa030a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 0x7fa02c1a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02be00640 1 -- 172.21.15.2:0/3100975865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 msgr2=0x7fa02c1a26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.811+0000 7fa02be00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 0x7fa02c1a26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa02be00640 1 -- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c075fd0 msgr2=0x7fa02c19e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa02be00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c075fd0 0x7fa02c19e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa02be00640 1 -- 172.21.15.2:0/3100975865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa018009560 con 0x7fa02c0756b0 2024-03-11T18:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa030a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 0x7fa02c1a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa02be00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00e9f0 tx=0x7fa01c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa028c00640 1 -- 172.21.15.2:0/3100975865 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c009720 con 0x7fa02c0756b0 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa028c00640 1 -- 172.21.15.2:0/3100975865 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa01c00cd80 con 0x7fa02c0756b0 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa028c00640 1 -- 172.21.15.2:0/3100975865 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c010690 con 0x7fa02c0756b0 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.812+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c1a3300 con 0x7fa02c0756b0 2024-03-11T18:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.813+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c1aab80 con 0x7fa02c0756b0 2024-03-11T18:39:20.818 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.816+0000 7fa028c00640 1 -- 172.21.15.2:0/3100975865 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fa01c0046c0 con 0x7fa02c0756b0 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.816+0000 7fa03577c740 1 Processor -- start 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.816+0000 7fa03577c740 1 -- start start 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 0x55d11783f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117840230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d11784b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1177fb620 con 0x55d117849330 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1177ad8c0 con 0x55d117840b20 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa03577c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1177aadf0 con 0x55d11783fdb0 2024-03-11T18:39:20.820 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa02be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117840230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa030a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 0x55d11783f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa030a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 0x55d11783f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38334/0 (socket says 172.21.15.2:38334) 2024-03-11T18:39:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa030a00640 1 -- 172.21.15.2:0/2353508154 learned_addr learned my addr 172.21.15.2:0/2353508154 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa02b400640 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d11784b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.821 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa028c00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 0x7fa0100a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.817+0000 7fa02be00640 1 -- 172.21.15.2:0/2353508154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 msgr2=0x55d11783f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa02be00640 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 0x55d11783f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa02be00640 1 -- 172.21.15.2:0/2353508154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 msgr2=0x55d11784b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa02be00640 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d11784b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa02be00640 1 -- 172.21.15.2:0/2353508154 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d117727b00 con 0x55d11783fdb0 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa028c00640 1 -- 172.21.15.2:0/3100975865 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa01c014070 con 0x7fa02c0756b0 2024-03-11T18:39:20.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa030a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 0x7fa0100a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa02be00640 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117840230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0cb2f0 tx=0x7fa01c004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=299c01bddbfd68e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa021200640 1 -- 172.21.15.2:0/2353508154 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c09a310 con 0x55d11783fdb0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa021200640 1 -- 172.21.15.2:0/2353508154 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa01c09a130 con 0x55d11783fdb0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.818+0000 7fa021200640 1 -- 172.21.15.2:0/2353508154 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c09e920 con 0x55d11783fdb0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 -- 172.21.15.2:0/2353508154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 msgr2=0x55d117840230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117840230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0cb2f0 tx=0x7fa01c004b30 comp rx=0 tx=0).stop 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa030a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 0x7fa0100a40d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa0240134c0 tx=0x7fa024004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 -- 172.21.15.2:0/2353508154 shutdown_connections 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d11784b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117840230 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 --2- 172.21.15.2:0/2353508154 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117840b20 0x55d11783f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 -- 172.21.15.2:0/2353508154 >> 172.21.15.2:0/2353508154 conn(0x55d1178074b0 msgr2=0x55d11783aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 -- 172.21.15.2:0/2353508154 shutdown_connections 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.819+0000 7fa03577c740 1 -- 172.21.15.2:0/2353508154 wait complete. 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 -- start start 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 Processor -- start 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d117846e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1177a65d0 con 0x55d117849330 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11774bb50 con 0x55d11783fdb0 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.820+0000 7fa03577c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11774b7f0 con 0x55d117840b20 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa03577c740 10 client.0 _ll_register_callbacks cb 0x55d11775f550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51546/0 (socket says 172.21.15.2:51546) 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38348/0 (socket says 172.21.15.2:38348) 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 -- 172.21.15.2:0/1556908278 learned_addr learned my addr 172.21.15.2:0/1556908278 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa030a00640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d117846e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 -- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 msgr2=0x55d1178463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 -- 172.21.15.2:0/1556908278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 msgr2=0x55d117846e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d117846e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 -- 172.21.15.2:0/1556908278 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c1aab80 con 0x55d117840b20 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02be00640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa02b400640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa018036040 tx=0x7fa01802c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f05edf1607bba8f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.821+0000 7fa015600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018004050 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa0180041b0 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d117778cd0 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa018004370 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa03577c740 10 client.25918 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d1177a4c50 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa018051080 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.822+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa01803c040 con 0x55d117840b20 2024-03-11T18:39:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 10 client.25918 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 20 client.25918 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178592e0 0x55d11785b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 4 client.25918 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d11779cce0 con 0x55d1178592e0 2024-03-11T18:39:20.831 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa02be00640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178592e0 0x55d11785b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa02be00640 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178592e0 0x55d11785b700 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f45e0606c19b6c server_cookie=4f2006ec42cf22a in_seq=0 out_seq=0 2024-03-11T18:39:20.833 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.823+0000 7fa015600640 10 client.25918 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.824+0000 7fa015600640 1 -- 172.21.15.2:0/1556908278 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d11779cce0 con 0x55d1178592e0 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.824+0000 7fa015600640 10 client.25918 handle_command_reply: tid=0 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.824+0000 7fa03577c740 1 client.25918 shutdown 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.824+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 msgr2=0x55d1178468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.824+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa018036040 tx=0x7fa01802c740 comp rx=0 tx=0).stop 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 shutdown_connections 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178592e0 0x55d11785b700 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d117846e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d1178468e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa015600640 0 client.25918 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 --2- 172.21.15.2:0/1556908278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 >> 172.21.15.2:0/1556908278 conn(0x55d1178074b0 msgr2=0x55d11783c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:20.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 shutdown_connections 2024-03-11T18:39:20.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 1 -- 172.21.15.2:0/1556908278 wait complete. 2024-03-11T18:39:20.836 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.825+0000 7fa03577c740 20 client.25918 trim_cache size 0 max 16384 2024-03-11T18:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.841+0000 7fa03577c740 1 Processor -- start 2024-03-11T18:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.841+0000 7fa03577c740 1 -- start start 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 0x55d1178500f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 0x55d117850630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1174eee60 con 0x55d117840b20 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11779cd10 con 0x55d117849330 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa03577c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d117855e20 con 0x55d11783fdb0 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 0x55d117850630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51548/0 (socket says 172.21.15.2:51548) 2024-03-11T18:39:20.845 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02be00640 1 -- 172.21.15.2:0/1630988915 learned_addr learned my addr 172.21.15.2:0/1630988915 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02b400640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 0x55d1178500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 -- 172.21.15.2:0/1630988915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 msgr2=0x55d117846c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 -- 172.21.15.2:0/1630988915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 msgr2=0x55d1178500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 0x55d1178500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa030a00640 1 -- 172.21.15.2:0/1630988915 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d117741530 con 0x55d117849330 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02be00640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.842+0000 7fa02b400640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 0x55d1178500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.843+0000 7fa030a00640 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 0x55d117850630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa024005fd0 tx=0x7fa024009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d29cdbf528c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.843+0000 7fa017e00640 1 -- 172.21.15.2:0/1630988915 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa024022050 con 0x55d117849330 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.843+0000 7fa017e00640 1 -- 172.21.15.2:0/1630988915 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa024023040 con 0x55d117849330 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.843+0000 7fa017e00640 1 -- 172.21.15.2:0/1630988915 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02401b8c0 con 0x55d117849330 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- 172.21.15.2:0/1630988915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 msgr2=0x55d117850630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 0x55d117850630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa024005fd0 tx=0x7fa024009f90 comp rx=0 tx=0).stop 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- 172.21.15.2:0/1630988915 shutdown_connections 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d117849330 0x55d117850630 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117840b20 0x55d1178500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 --2- 172.21.15.2:0/1630988915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d11783fdb0 0x55d117846c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- 172.21.15.2:0/1630988915 >> 172.21.15.2:0/1630988915 conn(0x55d1178074b0 msgr2=0x55d11783a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- 172.21.15.2:0/1630988915 shutdown_connections 2024-03-11T18:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- 172.21.15.2:0/1630988915 wait complete. 2024-03-11T18:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.844+0000 7fa03577c740 1 -- start start 2024-03-11T18:39:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 Processor -- start 2024-03-11T18:39:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d117842c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d1178431b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d117843810 con 0x55d117849330 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d117843980 con 0x55d11783fdb0 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d117843b40 con 0x55d117840b20 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa03577c740 10 client.0 _ll_register_callbacks cb 0x7fa018005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.845+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38372/0 (socket says 172.21.15.2:38372) 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 -- 172.21.15.2:0/2787042647 learned_addr learned my addr 172.21.15.2:0/2787042647 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02b400640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d117842c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa030a00640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d1178431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 -- 172.21.15.2:0/2787042647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 msgr2=0x55d117842c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d117842c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 -- 172.21.15.2:0/2787042647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 msgr2=0x55d1178431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d1178431b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 -- 172.21.15.2:0/2787042647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d117727b00 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa030a00640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d1178431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa02be00640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa01c098a80 tx=0x7fa01c0ba040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd29ddc99a9c5036 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa015600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c09a7d0 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7fa01c0cf070 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.846+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1177cbd30 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa01c09a310 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 10 client.25931 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fa024010540 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d1177cbd30 con 0x55d11783fdb0 2024-03-11T18:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa01c0a6070 con 0x55d11783fdb0 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 10 client.25931 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 10 client.25931 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 20 client.25931 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178b0c70 0x55d1178b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 4 client.25931 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.847+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d1174a6600 con 0x55d1178b0c70 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.848+0000 7fa02b400640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178b0c70 0x55d1178b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.848+0000 7fa02b400640 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178b0c70 0x55d1178b3090 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8537e3221b65aa5 server_cookie=3c828b12cae7addc in_seq=0 out_seq=0 2024-03-11T18:39:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:20.848+0000 7fa015600640 10 client.25931 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa015600640 1 -- 172.21.15.2:0/2787042647 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d1174a6600 con 0x55d1178b0c70 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa015600640 10 client.25931 handle_command_reply: tid=0 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 client.25931 shutdown 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 msgr2=0x55d1178503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa01c098a80 tx=0x7fa01c0ba040 comp rx=0 tx=0).stop 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 shutdown_connections 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d1178b0c70 0x55d1178b3090 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d117849330 0x55d1178431b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d117840b20 0x55d117842c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 --2- 172.21.15.2:0/2787042647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d11783fdb0 0x55d1178503d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.084 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.080+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 >> 172.21.15.2:0/2787042647 conn(0x55d1178074b0 msgr2=0x55d117844bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.081+0000 7fa015600640 0 client.25931 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.081+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 shutdown_connections 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.081+0000 7fa03577c740 1 -- 172.21.15.2:0/2787042647 wait complete. 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.081+0000 7fa03577c740 20 client.25931 trim_cache size 0 max 16384 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:39:21.085 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 msgr2=0x7fa0100a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 0x7fa0100a40d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa0240134c0 tx=0x7fa024004a40 comp rx=0 tx=0).stop 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 msgr2=0x7fa02c19de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00e9f0 tx=0x7fa01c00edf0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.086 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 shutdown_connections 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa02c111ff0 0x7fa02c1a26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa0100a1c10 0x7fa0100a40d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa02c075fd0 0x7fa02c19e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 --2- 172.21.15.2:0/3100975865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa02c0756b0 0x7fa02c19de30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 >> 172.21.15.2:0/3100975865 conn(0x7fa02c0fee00 msgr2=0x7fa02c113b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 shutdown_connections 2024-03-11T18:39:21.087 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:21.082+0000 7fa032a00640 1 -- 172.21.15.2:0/3100975865 wait complete. 2024-03-11T18:39:21.408 INFO:tasks.workunit.client.0.smithi002.stdout: 1 426158 8.41 MB/sec execute 141 sec latency 114.421 ms 2024-03-11T18:39:22.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:22 smithi165 ceph-mon[36634]: pgmap v172: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:22.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:22 smithi023 ceph-mon[36406]: pgmap v172: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:22.409 INFO:tasks.workunit.client.0.smithi002.stdout: 1 427341 8.39 MB/sec execute 142 sec latency 30.415 ms 2024-03-11T18:39:22.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:22 smithi002 ceph-mon[31328]: pgmap v172: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:23.409 INFO:tasks.workunit.client.0.smithi002.stdout: 1 429431 8.42 MB/sec execute 143 sec latency 67.426 ms 2024-03-11T18:39:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:24 smithi023 ceph-mon[36406]: pgmap v173: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:24.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:24 smithi165 ceph-mon[36634]: pgmap v173: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:24.409 INFO:tasks.workunit.client.0.smithi002.stdout: 1 430225 8.37 MB/sec execute 144 sec latency 20.480 ms 2024-03-11T18:39:24.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:24 smithi002 ceph-mon[31328]: pgmap v173: 129 pgs: 129 active+clean; 66 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-03-11T18:39:25.409 INFO:tasks.workunit.client.0.smithi002.stdout: 1 432416 8.40 MB/sec execute 145 sec latency 13.108 ms 2024-03-11T18:39:26.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:26 smithi165 ceph-mon[36634]: pgmap v174: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 371 op/s 2024-03-11T18:39:26.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:26 smithi023 ceph-mon[36406]: pgmap v174: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 371 op/s 2024-03-11T18:39:26.409 INFO:tasks.workunit.client.0.smithi002.stdout: 1 433510 8.39 MB/sec execute 146 sec latency 18.579 ms 2024-03-11T18:39:26.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:26 smithi002 ceph-mon[31328]: pgmap v174: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 371 op/s 2024-03-11T18:39:27.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 435244 8.38 MB/sec execute 147 sec latency 18.686 ms 2024-03-11T18:39:28.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:28 smithi165 ceph-mon[36634]: pgmap v175: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-03-11T18:39:28.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:28 smithi023 ceph-mon[36406]: pgmap v175: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-03-11T18:39:28.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 436865 8.40 MB/sec execute 148 sec latency 28.856 ms 2024-03-11T18:39:28.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:28 smithi002 ceph-mon[31328]: pgmap v175: 129 pgs: 129 active+clean; 62 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-03-11T18:39:29.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 438728 8.39 MB/sec execute 149 sec latency 15.487 ms 2024-03-11T18:39:30.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:30 smithi165 ceph-mon[36634]: pgmap v176: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 368 op/s 2024-03-11T18:39:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:30 smithi023 ceph-mon[36406]: pgmap v176: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 368 op/s 2024-03-11T18:39:30.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 440449 8.42 MB/sec execute 150 sec latency 14.895 ms 2024-03-11T18:39:30.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:30 smithi002 ceph-mon[31328]: pgmap v176: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 368 op/s 2024-03-11T18:39:31.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 442304 8.41 MB/sec execute 151 sec latency 15.965 ms 2024-03-11T18:39:32.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:32 smithi165 ceph-mon[36634]: pgmap v177: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:32 smithi023 ceph-mon[36406]: pgmap v177: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:32.410 INFO:tasks.workunit.client.0.smithi002.stdout: 1 443937 8.43 MB/sec execute 152 sec latency 17.286 ms 2024-03-11T18:39:32.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:32 smithi002 ceph-mon[31328]: pgmap v177: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:33.411 INFO:tasks.workunit.client.0.smithi002.stdout: 1 445384 8.41 MB/sec execute 153 sec latency 54.000 ms 2024-03-11T18:39:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 -- 172.21.15.2:0/1106331635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da0101d80 msgr2=0x7f9da0105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 --2- 172.21.15.2:0/1106331635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da0101d80 0x7f9da0105dc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009930 tx=0x7f9d8c02bc20 comp rx=0 tx=0).stop 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 -- 172.21.15.2:0/1106331635 shutdown_connections 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 --2- 172.21.15.2:0/1106331635 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da01064a0 0x7f9da010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 --2- 172.21.15.2:0/1106331635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da0101d80 0x7f9da0105dc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 --2- 172.21.15.2:0/1106331635 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da01017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 -- 172.21.15.2:0/1106331635 >> 172.21.15.2:0/1106331635 conn(0x7f9da00fcc10 msgr2=0x7f9da00ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 -- 172.21.15.2:0/1106331635 shutdown_connections 2024-03-11T18:39:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.235+0000 7f9da4e00640 1 -- 172.21.15.2:0/1106331635 wait complete. 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 Processor -- start 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 -- start start 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 0x7f9da019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da0199860 0x7f9da0199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da019a250 con 0x7f9da01013d0 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da019a3c0 con 0x7f9da01064a0 2024-03-11T18:39:34.251 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9da4e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da019a580 con 0x7f9da0199860 2024-03-11T18:39:34.252 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.236+0000 7f9d9de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 0x7f9da019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59108/0 (socket says 172.21.15.2:59108) 2024-03-11T18:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 -- 172.21.15.2:0/2067338866 learned_addr learned my addr 172.21.15.2:0/2067338866 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da0199860 0x7f9da0199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 -- 172.21.15.2:0/2067338866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da0199860 msgr2=0x7f9da0199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da0199860 0x7f9da0199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 -- 172.21.15.2:0/2067338866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 msgr2=0x7f9da019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 0x7f9da019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 -- 172.21.15.2:0/2067338866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d90009490 con 0x7f9da01013d0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 0x7f9da019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.237+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000ea60 tx=0x7f9d9000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.238+0000 7f9d96a00640 1 -- 172.21.15.2:0/2067338866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9000cde0 con 0x7f9da01013d0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.238+0000 7f9d96a00640 1 -- 172.21.15.2:0/2067338866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d900035f0 con 0x7f9da01013d0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.238+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c009560 con 0x7f9da01013d0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.238+0000 7f9d96a00640 1 -- 172.21.15.2:0/2067338866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90010690 con 0x7f9da01013d0 2024-03-11T18:39:34.263 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.238+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da01a68c0 con 0x7f9da01013d0 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.241+0000 7f9d96a00640 1 -- 172.21.15.2:0/2067338866 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f9d90003af0 con 0x7f9da01013d0 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.241+0000 7f9da7b6a740 1 Processor -- start 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 -- start start 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 0x564d1488c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d14897720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d14847620 con 0x564d1488bdb0 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d147f98c0 con 0x564d14895330 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.242+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d147f6df0 con 0x564d1488cb20 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39066/0 (socket says 172.21.15.2:39066) 2024-03-11T18:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 -- 172.21.15.2:0/2569900911 learned_addr learned my addr 172.21.15.2:0/2569900911 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 0x564d1488c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d14897720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 -- 172.21.15.2:0/2569900911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 msgr2=0x564d14897720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d14897720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 -- 172.21.15.2:0/2569900911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 msgr2=0x564d1488c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 0x564d1488c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 -- 172.21.15.2:0/2569900911 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d14773b00 con 0x564d1488cb20 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d14897720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 0x564d1488c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:34.265 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d96a00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 0x7f9d840a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d96a00640 1 -- 172.21.15.2:0/2067338866 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9d90014070 con 0x7f9da01013d0 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.243+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009f50 tx=0x7f9d980118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aae76a73d2609c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.244+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 0x7f9d840a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.244+0000 7f9d8b400640 1 -- 172.21.15.2:0/2569900911 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98004d20 con 0x564d1488cb20 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.244+0000 7f9d8b400640 1 -- 172.21.15.2:0/2569900911 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d98004e80 con 0x564d1488cb20 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.244+0000 7f9d8b400640 1 -- 172.21.15.2:0/2569900911 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98005280 con 0x564d1488cb20 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2569900911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 msgr2=0x564d1488b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9d9de00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 0x7f9d840a4050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x564d1488d280 tx=0x7f9d900ba040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009f50 tx=0x7f9d980118d0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2569900911 shutdown_connections 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d14897720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488bdb0 0x564d1488c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2569900911 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488cb20 0x564d1488b870 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2569900911 >> 172.21.15.2:0/2569900911 conn(0x564d148534b0 msgr2=0x564d14886aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2569900911 shutdown_connections 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2569900911 wait complete. 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.245+0000 7f9da7b6a740 1 -- start start 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 Processor -- start 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d148928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d14895330 0x564d14892e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d147f25d0 con 0x564d1488cb20 2024-03-11T18:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d14797b50 con 0x564d1488bdb0 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d147977f0 con 0x564d14895330 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9da7b6a740 10 client.0 _ll_register_callbacks cb 0x564d147ab550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50738/0 (socket says 172.21.15.2:50738) 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9d9e800640 1 -- 172.21.15.2:0/3336772690 learned_addr learned my addr 172.21.15.2:0/3336772690 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.246+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d148928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d14895330 0x564d14892e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 -- 172.21.15.2:0/3336772690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d14895330 msgr2=0x564d14892e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d14895330 0x564d14892e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 -- 172.21.15.2:0/3336772690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 msgr2=0x564d148928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d148928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 -- 172.21.15.2:0/3336772690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d98009c00 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9d98004670 tx=0x7f9d98004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbadb69644a8eac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98005200 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.247+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d147c4cd0 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9d98005360 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d98005520 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 10 client.25943 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564d147f0c50 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d98036080 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9d98027ad0 con 0x564d1488bdb0 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 10 client.25943 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 10 client.25943 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 20 client.25943 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148a52f0 0x564d148a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 4 client.25943 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.248+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d147e8ce0 con 0x564d148a52f0 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.249+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148a52f0 0x564d148a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.249+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148a52f0 0x564d148a76e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a95fae48ebf2920 server_cookie=591e10992530bae2 in_seq=0 out_seq=0 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.249+0000 7f9d7fe00640 10 client.25943 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3336772690 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d147e8ce0 con 0x564d148a52f0 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9d7fe00640 10 client.25943 handle_command_reply: tid=0 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 client.25943 shutdown 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 msgr2=0x564d14892380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9d98004670 tx=0x7f9d98004770 comp rx=0 tx=0).stop 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 shutdown_connections 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148a52f0 0x564d148a76e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d14895330 0x564d14892e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d148928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9d7fe00640 0 client.25943 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3336772690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488bdb0 0x564d14892380 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 >> 172.21.15.2:0/3336772690 conn(0x564d148534b0 msgr2=0x564d148887f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.250+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 shutdown_connections 2024-03-11T18:39:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.251+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3336772690 wait complete. 2024-03-11T18:39:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.251+0000 7f9da7b6a740 20 client.25943 trim_cache size 0 max 16384 2024-03-11T18:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.266+0000 7f9da7b6a740 1 Processor -- start 2024-03-11T18:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.266+0000 7f9da7b6a740 1 -- start start 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d14894ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 0x564d14892e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d1453ae60 con 0x564d14895330 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d148a50f0 con 0x564d1488cb20 2024-03-11T18:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d14887ad0 con 0x564d1488bdb0 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50744/0 (socket says 172.21.15.2:50744) 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 -- 172.21.15.2:0/2164292191 learned_addr learned my addr 172.21.15.2:0/2164292191 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d14894ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 0x564d14892e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 -- 172.21.15.2:0/2164292191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 msgr2=0x564d14894ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d14894ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.267+0000 7f9d9d400640 1 -- 172.21.15.2:0/2164292191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 msgr2=0x564d14892e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 0x564d14892e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d9d400640 1 -- 172.21.15.2:0/2164292191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d1478d530 con 0x564d1488cb20 2024-03-11T18:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d9e800640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 0x564d14892e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d9d400640 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c000c00 tx=0x7f9d8c005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9953e89042800bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d8aa00640 1 -- 172.21.15.2:0/2164292191 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c048be0 con 0x564d1488cb20 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d8aa00640 1 -- 172.21.15.2:0/2164292191 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d8c048d40 con 0x564d1488cb20 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.268+0000 7f9d8aa00640 1 -- 172.21.15.2:0/2164292191 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c048600 con 0x564d1488cb20 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2164292191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 msgr2=0x564d148928d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c000c00 tx=0x7f9d8c005d90 comp rx=0 tx=0).stop 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2164292191 shutdown_connections 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d14895330 0x564d14892e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d1488cb20 0x564d148928d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/2164292191 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d14894ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2164292191 >> 172.21.15.2:0/2164292191 conn(0x564d148534b0 msgr2=0x564d14890c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.280 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2164292191 shutdown_connections 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.269+0000 7f9da7b6a740 1 -- 172.21.15.2:0/2164292191 wait complete. 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 -- start start 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 Processor -- start 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d1489ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d1489b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d148fb960 con 0x564d1488cb20 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d148fbad0 con 0x564d14895330 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.270+0000 7f9da7b6a740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d148fbc90 con 0x564d1488bdb0 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9da7b6a740 10 client.0 _ll_register_callbacks cb 0x7f9d98009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d1489b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59142/0 (socket says 172.21.15.2:59142) 2024-03-11T18:39:34.281 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 -- 172.21.15.2:0/3707147086 learned_addr learned my addr 172.21.15.2:0/3707147086 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d1489ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 -- 172.21.15.2:0/3707147086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 msgr2=0x564d1489ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d1489ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 -- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 msgr2=0x564d1489b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d1489b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9d400640 1 -- 172.21.15.2:0/3707147086 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da010e880 con 0x564d1488cb20 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.271+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d1489b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9d9d400640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009230 tx=0x7f9d8c048cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17db3b9a14df62a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9d7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c02c850 con 0x564d1488cb20 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f9d8c045040 con 0x564d1488cb20 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d14817d30 con 0x564d1488cb20 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d8c02c9b0 con 0x564d1488cb20 2024-03-11T18:39:34.282 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9da7b6a740 10 client.15816 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.272+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564d147b32c0 con 0x564d1488cb20 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d14817d30 con 0x564d1488cb20 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9d8c03c040 con 0x564d1488cb20 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 10 client.15816 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 20 client.15816 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148fd050 0x564d148ff440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 4 client.15816 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564d144f2600 con 0x564d148fd050 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.273+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148fd050 0x564d148ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.274+0000 7f9d9e800640 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148fd050 0x564d148ff440 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=270073fe817d6e5f server_cookie=b0237b92928071b6 in_seq=0 out_seq=0 2024-03-11T18:39:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.274+0000 7f9d7fe00640 10 client.15816 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:34.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:34 smithi023 ceph-mon[36406]: pgmap v178: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:34.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:34 smithi165 ceph-mon[36634]: pgmap v178: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:34.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 447195 8.44 MB/sec execute 154 sec latency 64.534 ms 2024-03-11T18:39:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9d7fe00640 1 -- 172.21.15.2:0/3707147086 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564d144f2600 con 0x564d148fd050 2024-03-11T18:39:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9d7fe00640 10 client.15816 handle_command_reply: tid=0 2024-03-11T18:39:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 client.15816 shutdown 2024-03-11T18:39:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 msgr2=0x564d1489b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c009230 tx=0x7f9d8c048cd0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 shutdown_connections 2024-03-11T18:39:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x564d148fd050 0x564d148ff440 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d14895330 0x564d1489b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x564d1488cb20 0x564d1489b390 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 --2- 172.21.15.2:0/3707147086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x564d1488bdb0 0x564d1489ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.491+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 >> 172.21.15.2:0/3707147086 conn(0x564d148534b0 msgr2=0x564d148854b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.492+0000 7f9d7fe00640 0 client.15816 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.492+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 shutdown_connections 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.492+0000 7f9da7b6a740 1 -- 172.21.15.2:0/3707147086 wait complete. 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.492+0000 7f9da7b6a740 20 client.15816 trim_cache size 0 max 16384 2024-03-11T18:39:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:39:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:39:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:39:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:39:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 msgr2=0x7f9d840a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 0x7f9d840a4050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x564d1488d280 tx=0x7f9d900ba040 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 msgr2=0x7f9da019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9d9000ea60 tx=0x7f9d9000ef20 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 shutdown_connections 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9da0199860 0x7f9da0199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9da01064a0 0x7f9da019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9d840a1b90 0x7f9d840a4050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 --2- 172.21.15.2:0/2067338866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9da01013d0 0x7f9da019f6b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.493+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 >> 172.21.15.2:0/2067338866 conn(0x7f9da00fcc10 msgr2=0x7f9da00fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.494+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 shutdown_connections 2024-03-11T18:39:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:34.494+0000 7f9da4e00640 1 -- 172.21.15.2:0/2067338866 wait complete. 2024-03-11T18:39:34.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:34 smithi002 ceph-mon[31328]: pgmap v178: 129 pgs: 129 active+clean; 67 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 278 op/s 2024-03-11T18:39:35.411 INFO:tasks.workunit.client.0.smithi002.stdout: 1 448178 8.40 MB/sec execute 155 sec latency 118.552 ms 2024-03-11T18:39:36.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:36 smithi165 ceph-mon[36634]: pgmap v179: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 364 op/s 2024-03-11T18:39:36.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:36 smithi023 ceph-mon[36406]: pgmap v179: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 364 op/s 2024-03-11T18:39:36.411 INFO:tasks.workunit.client.0.smithi002.stdout: 1 450504 8.45 MB/sec execute 156 sec latency 12.998 ms 2024-03-11T18:39:36.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:36 smithi002 ceph-mon[31328]: pgmap v179: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.9 MiB/s wr, 364 op/s 2024-03-11T18:39:37.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 451409 8.41 MB/sec execute 157 sec latency 15.852 ms 2024-03-11T18:39:38.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:38 smithi165 ceph-mon[36634]: pgmap v180: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-03-11T18:39:38.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:38 smithi023 ceph-mon[36406]: pgmap v180: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-03-11T18:39:38.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 453481 8.43 MB/sec execute 158 sec latency 46.532 ms 2024-03-11T18:39:38.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:38 smithi002 ceph-mon[31328]: pgmap v180: 129 pgs: 129 active+clean; 71 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 166 op/s 2024-03-11T18:39:39.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 454840 8.42 MB/sec execute 159 sec latency 16.924 ms 2024-03-11T18:39:40.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:40 smithi165 ceph-mon[36634]: pgmap v181: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-11T18:39:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:40 smithi023 ceph-mon[36406]: pgmap v181: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-11T18:39:40.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 456931 8.44 MB/sec execute 160 sec latency 54.499 ms 2024-03-11T18:39:40.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:40 smithi002 ceph-mon[31328]: pgmap v181: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-11T18:39:41.412 INFO:tasks.workunit.client.0.smithi002.stdout: 1 458344 8.44 MB/sec execute 161 sec latency 59.695 ms 2024-03-11T18:39:42.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:42 smithi165 ceph-mon[36634]: pgmap v182: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:42.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:42 smithi023 ceph-mon[36406]: pgmap v182: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:42.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 459372 8.52 MB/sec execute 162 sec latency 59.569 ms 2024-03-11T18:39:42.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:42 smithi002 ceph-mon[31328]: pgmap v182: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:43.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 461328 8.54 MB/sec execute 163 sec latency 56.269 ms 2024-03-11T18:39:44.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:44 smithi165 ceph-mon[36634]: pgmap v183: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:44 smithi023 ceph-mon[36406]: pgmap v183: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:44.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 462641 8.53 MB/sec execute 164 sec latency 16.222 ms 2024-03-11T18:39:44.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:44 smithi002 ceph-mon[31328]: pgmap v183: 129 pgs: 129 active+clean; 64 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 265 op/s 2024-03-11T18:39:45.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 464306 8.52 MB/sec execute 165 sec latency 66.023 ms 2024-03-11T18:39:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:46 smithi165 ceph-mon[36634]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-11T18:39:46.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:46 smithi023 ceph-mon[36406]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-11T18:39:46.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 465879 8.54 MB/sec execute 166 sec latency 14.080 ms 2024-03-11T18:39:46.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:46 smithi002 ceph-mon[31328]: pgmap v184: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-11T18:39:47.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 467138 8.52 MB/sec execute 167 sec latency 20.058 ms 2024-03-11T18:39:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 -- 172.21.15.2:0/3660253185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58068be0 msgr2=0x7f6e58068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 --2- 172.21.15.2:0/3660253185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58068be0 0x7f6e58068fc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009860 tx=0x7f6e4c02bb70 comp rx=0 tx=0).stop 2024-03-11T18:39:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 -- 172.21.15.2:0/3660253185 shutdown_connections 2024-03-11T18:39:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 --2- 172.21.15.2:0/3660253185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58106370 0x7f6e5810deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.653 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 --2- 172.21.15.2:0/3660253185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58069590 0x7f6e58105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 --2- 172.21.15.2:0/3660253185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58068be0 0x7f6e58068fc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 -- 172.21.15.2:0/3660253185 >> 172.21.15.2:0/3660253185 conn(0x7f6e580759e0 msgr2=0x7f6e58075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 -- 172.21.15.2:0/3660253185 shutdown_connections 2024-03-11T18:39:47.654 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.651+0000 7f6e5e800640 1 -- 172.21.15.2:0/3660253185 wait complete. 2024-03-11T18:39:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 Processor -- start 2024-03-11T18:39:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 -- start start 2024-03-11T18:39:47.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58106370 0x7f6e58100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 0x7f6e581a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.652+0000 7f6e5e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581a5de0 con 0x7f6e58100c50 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5e800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581a5f50 con 0x7f6e58068be0 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5e800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e581a6110 con 0x7f6e58106370 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50714/0 (socket says 172.21.15.2:50714) 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 -- 172.21.15.2:0/1885671698 learned_addr learned my addr 172.21.15.2:0/1885671698 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5de00640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 0x7f6e581a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58106370 0x7f6e58100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 -- 172.21.15.2:0/1885671698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58106370 msgr2=0x7f6e58100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58106370 0x7f6e58100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 -- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 msgr2=0x7f6e581a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 0x7f6e581a5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 -- 172.21.15.2:0/1885671698 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e48009560 con 0x7f6e58068be0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5de00640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 0x7f6e581a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.653+0000 7f6e5d400640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c037040 tx=0x7f6e4c0045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.654+0000 7f6e56000640 1 -- 172.21.15.2:0/1885671698 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0349d0 con 0x7f6e58068be0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.654+0000 7f6e56000640 1 -- 172.21.15.2:0/1885671698 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c02c9f0 con 0x7f6e58068be0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.654+0000 7f6e56000640 1 -- 172.21.15.2:0/1885671698 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0035f0 con 0x7f6e58068be0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.654+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e50009840 con 0x7f6e58068be0 2024-03-11T18:39:47.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.654+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c009490 con 0x7f6e58068be0 2024-03-11T18:39:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.657+0000 7f6e56000640 1 -- 172.21.15.2:0/1885671698 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f6e4c03b040 con 0x7f6e58068be0 2024-03-11T18:39:47.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 Processor -- start 2024-03-11T18:39:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 -- start start 2024-03-11T18:39:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc7d60 0x55c486cc6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc6ff0 0x55c486cc7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486c82620 con 0x55c486cc7d60 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486c348c0 con 0x55c486cd0570 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.658+0000 7f6e62a11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486c31df0 con 0x55c486cc6ff0 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc7d60 0x55c486cc6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50716/0 (socket says 172.21.15.2:50716) 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3484173900 learned_addr learned my addr 172.21.15.2:0/3484173900 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5d400640 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc6ff0 0x55c486cc7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3484173900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc6ff0 msgr2=0x55c486cc7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc6ff0 0x55c486cc7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3484173900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc7d60 msgr2=0x55c486cc6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc7d60 0x55c486cc6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3484173900 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c486baeb00 con 0x55c486cd0570 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e56000640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 0x7f6e400a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009d00 tx=0x7f6e480117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c45f2cfd77afb8e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.659+0000 7f6e56000640 1 -- 172.21.15.2:0/1885671698 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e4c0ef650 con 0x7f6e58068be0 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.660+0000 7f6e46600640 1 -- 172.21.15.2:0/3484173900 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48005020 con 0x55c486cd0570 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.660+0000 7f6e46600640 1 -- 172.21.15.2:0/3484173900 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e48005660 con 0x55c486cd0570 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.660+0000 7f6e46600640 1 -- 172.21.15.2:0/3484173900 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480127b0 con 0x55c486cd0570 2024-03-11T18:39:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.660+0000 7f6e5de00640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 0x7f6e400a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 -- 172.21.15.2:0/3484173900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 msgr2=0x55c486cd2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e5de00640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 0x7f6e400a40f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e50005fd0 tx=0x7f6e50009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009d00 tx=0x7f6e480117b0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 -- 172.21.15.2:0/3484173900 shutdown_connections 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd2960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc6ff0 0x55c486cc7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3484173900 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc7d60 0x55c486cc6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 -- 172.21.15.2:0/3484173900 >> 172.21.15.2:0/3484173900 conn(0x55c486c8e4b0 msgr2=0x55c486cc1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 -- 172.21.15.2:0/3484173900 shutdown_connections 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.661+0000 7f6e62a11740 1 -- 172.21.15.2:0/3484173900 wait complete. 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.662+0000 7f6e62a11740 1 -- start start 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.662+0000 7f6e62a11740 1 Processor -- start 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.662+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.662+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.662+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 0x55c486cce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e62a11740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486c2d5d0 con 0x55c486cd0570 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e62a11740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486bd2b50 con 0x55c486cc6ff0 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e62a11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486bd27f0 con 0x55c486cc7d60 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e62a11740 10 client.0 _ll_register_callbacks cb 0x55c486be6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50728/0 (socket says 172.21.15.2:50728) 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:56380/0 (socket says 172.21.15.2:56380) 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 -- 172.21.15.2:0/2632303034 learned_addr learned my addr 172.21.15.2:0/2632303034 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5de00640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 0x55c486cce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 -- 172.21.15.2:0/2632303034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 msgr2=0x55c486ccdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 -- 172.21.15.2:0/2632303034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 msgr2=0x55c486cce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 0x55c486cce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5d400640 1 -- 172.21.15.2:0/2632303034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e48009960 con 0x55c486cc6ff0 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.663+0000 7f6e5de00640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 0x55c486cce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e5d400640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c02c180 tx=0x7f6e4c032f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc47215212a5ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c0ece20 con 0x55c486cc6ff0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e4c0ef020 con 0x55c486cc6ff0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c486bffcd0 con 0x55c486cc6ff0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e4c0f5030 con 0x55c486cc6ff0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e62a11740 10 client.25961 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c486c2bc50 con 0x55c486cc6ff0 2024-03-11T18:39:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.664+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e4c0d3020 con 0x55c486cc6ff0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6e4c0c42a0 con 0x55c486cc6ff0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 10 client.25961 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 10 client.25961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 20 client.25961 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486ce02e0 0x55c486ce26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 4 client.25961 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c486c23ce0 con 0x55c486ce02e0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.665+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486ce02e0 0x55c486ce26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.666+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486ce02e0 0x55c486ce26d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d919a05d5e7ba2b5 server_cookie=f2c9bdd1ea771fe1 in_seq=0 out_seq=0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.666+0000 7f6e3aa00640 10 client.25961 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.667+0000 7f6e3aa00640 1 -- 172.21.15.2:0/2632303034 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c486c23ce0 con 0x55c486ce02e0 2024-03-11T18:39:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.667+0000 7f6e3aa00640 10 client.25961 handle_command_reply: tid=0 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.667+0000 7f6e62a11740 1 client.25961 shutdown 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.667+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 msgr2=0x55c486ccd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.667+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c02c180 tx=0x7f6e4c032f00 comp rx=0 tx=0).stop 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 shutdown_connections 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486ce02e0 0x55c486ce26d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cd0570 0x55c486cce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e3aa00640 0 client.25961 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 --2- 172.21.15.2:0/2632303034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc6ff0 0x55c486ccd5b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 >> 172.21.15.2:0/2632303034 conn(0x55c486c8e4b0 msgr2=0x55c486cc3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 shutdown_connections 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 1 -- 172.21.15.2:0/2632303034 wait complete. 2024-03-11T18:39:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.668+0000 7f6e62a11740 20 client.25961 trim_cache size 0 max 16384 2024-03-11T18:39:47.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.684+0000 7f6e62a11740 1 Processor -- start 2024-03-11T18:39:47.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 -- start start 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486ccfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486975e60 con 0x55c486cc6ff0 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486cc2f10 con 0x55c486cd0570 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e62a11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486ce00e0 con 0x55c486cc7d60 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:56390/0 (socket says 172.21.15.2:56390) 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486ccfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3311546480 learned_addr learned my addr 172.21.15.2:0/3311546480 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.685+0000 7f6e5de00640 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.689 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3311546480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 msgr2=0x55c486cd7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3311546480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 msgr2=0x55c486ccfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486ccfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3311546480 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c486bc8530 con 0x55c486cc7d60 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5de00640 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55c486cc7a90 tx=0x7f6e48009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a671694b25ac8899 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e45c00640 1 -- 172.21.15.2:0/3311546480 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480049d0 con 0x55c486cc7d60 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e45c00640 1 -- 172.21.15.2:0/3311546480 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e48004bb0 con 0x55c486cc7d60 2024-03-11T18:39:47.690 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.686+0000 7f6e45c00640 1 -- 172.21.15.2:0/3311546480 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48005d10 con 0x55c486cc7d60 2024-03-11T18:39:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 -- 172.21.15.2:0/3311546480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 msgr2=0x55c486ccde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55c486cc7a90 tx=0x7f6e48009260 comp rx=0 tx=0).stop 2024-03-11T18:39:47.693 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 -- 172.21.15.2:0/3311546480 shutdown_connections 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cd0570 0x55c486cd7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cc7d60 0x55c486ccde20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3311546480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486ccfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 -- 172.21.15.2:0/3311546480 >> 172.21.15.2:0/3311546480 conn(0x55c486c8e4b0 msgr2=0x55c486cc1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 -- 172.21.15.2:0/3311546480 shutdown_connections 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.687+0000 7f6e62a11740 1 -- 172.21.15.2:0/3311546480 wait complete. 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 -- start start 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 Processor -- start 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486cd7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cd0570 0x55c486cd41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486cd4880 con 0x55c486cc6ff0 2024-03-11T18:39:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486cd49f0 con 0x55c486cc7d60 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c486d367b0 con 0x55c486cd0570 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.688+0000 7f6e62a11740 10 client.0 _ll_register_callbacks cb 0x7f6e4c0f5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486cd7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50748/0 (socket says 172.21.15.2:50748) 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3066982333 learned_addr learned my addr 172.21.15.2:0/3066982333 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5de00640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cd0570 0x55c486cd41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3066982333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cd0570 msgr2=0x55c486cd41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.695 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cd0570 0x55c486cd41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3066982333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 msgr2=0x55c486cd7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486cd7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 -- 172.21.15.2:0/3066982333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c486baeb00 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e5ca00640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009c50 tx=0x7f6e480092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c93448c9ea75d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480056e0 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.689+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e48009e70 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c48692d600 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e48004bb0 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 10 client.25967 fetch_fsmap learned FSMap version 27 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c486bee2c0 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c48692d600 con 0x55c486cc7d60 2024-03-11T18:39:47.696 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6e48033070 con 0x55c486cc7d60 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 10 client.25967 fetch_fsmap finished waiting for FSMap version 27 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 10 client.25967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 20 client.25967 populate_metadata read hostname 'smithi002' 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486d37c90 0x55c486d3a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 4 client.25967 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.690+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c48697e1b0 con 0x55c486d37c90 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.691+0000 7f6e5d400640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486d37c90 0x55c486d3a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.691+0000 7f6e5d400640 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486d37c90 0x55c486d3a0b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfaf41f5d76d690b server_cookie=550ee853b7338221 in_seq=0 out_seq=0 2024-03-11T18:39:47.697 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.691+0000 7f6e3aa00640 10 client.25967 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e3aa00640 1 -- 172.21.15.2:0/3066982333 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c48697e1b0 con 0x55c486d37c90 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e3aa00640 10 client.25967 handle_command_reply: tid=0 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 client.25967 shutdown 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 msgr2=0x55c486cd3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009c50 tx=0x7f6e480092d0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 shutdown_connections 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55c486d37c90 0x55c486d3a0b0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55c486cd0570 0x55c486cd41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c486cc7d60 0x55c486cd3ba0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 --2- 172.21.15.2:0/3066982333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55c486cc6ff0 0x55c486cd7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 >> 172.21.15.2:0/3066982333 conn(0x55c486c8e4b0 msgr2=0x55c486ccbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e3aa00640 0 client.25967 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 shutdown_connections 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 1 -- 172.21.15.2:0/3066982333 wait complete. 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.955+0000 7f6e62a11740 20 client.25967 trim_cache size 0 max 16384 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:39:47.959 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:39:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:39:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:39:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 msgr2=0x7f6e400a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 0x7f6e400a40f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e50005fd0 tx=0x7f6e50009500 comp rx=0 tx=0).stop 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 msgr2=0x7f6e581001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c037040 tx=0x7f6e4c0045b0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 shutdown_connections 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e400a1c30 0x7f6e400a40f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e58100c50 0x7f6e581a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e58106370 0x7f6e58100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 --2- 172.21.15.2:0/1885671698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e58068be0 0x7f6e581001d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:39:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.957+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 >> 172.21.15.2:0/1885671698 conn(0x7f6e580759e0 msgr2=0x7f6e580fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:39:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.958+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 shutdown_connections 2024-03-11T18:39:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:39:47.958+0000 7f6e5e800640 1 -- 172.21.15.2:0/1885671698 wait complete. 2024-03-11T18:39:48.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:48 smithi165 ceph-mon[36634]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:39:48.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:48 smithi023 ceph-mon[36406]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:39:48.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 468795 8.52 MB/sec execute 168 sec latency 163.976 ms 2024-03-11T18:39:48.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:48 smithi002 ceph-mon[31328]: pgmap v185: 129 pgs: 129 active+clean; 68 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:39:48.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:39:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setuser ceph since I am not root 2024-03-11T18:39:48.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:39:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setgroup ceph since I am not root 2024-03-11T18:39:48.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:39:48 smithi165 ceph-mds[67373]: -- 172.21.15.165:0/2401880528 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55da71c8c1a0 con 0x55da71c62800 2024-03-11T18:39:48.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:39:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: starting mds.c at 2024-03-11T18:39:49.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:49 smithi165 ceph-mon[36634]: daemon mds.c restarted 2024-03-11T18:39:49.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:49 smithi023 ceph-mon[36406]: daemon mds.c restarted 2024-03-11T18:39:49.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 469715 8.50 MB/sec execute 169 sec latency 58.252 ms 2024-03-11T18:39:49.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:49 smithi002 ceph-mon[31328]: daemon mds.c restarted 2024-03-11T18:39:50.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-03-11T18:39:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T18:39:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:39:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6836/1979460197,v1:172.21.15.165:6837/1979460197] up:boot 2024-03-11T18:39:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:39:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:50 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6836/1979460197,v1:172.21.15.165:6837/1979460197] up:boot 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:39:50.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:50 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:39:50.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 471343 8.49 MB/sec execute 170 sec latency 24.162 ms 2024-03-11T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: pgmap v186: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-03-11T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: osdmap e97: 12 total, 12 up, 12 in 2024-03-11T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:39:50.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6836/1979460197,v1:172.21.15.165:6837/1979460197] up:boot 2024-03-11T18:39:50.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:39:50.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:50 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:39:51.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 472799 8.51 MB/sec execute 171 sec latency 21.809 ms 2024-03-11T18:39:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:52 smithi023 ceph-mon[36406]: pgmap v188: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:52 smithi165 ceph-mon[36634]: pgmap v188: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:52.413 INFO:tasks.workunit.client.0.smithi002.stdout: 1 473598 8.47 MB/sec execute 172 sec latency 18.382 ms 2024-03-11T18:39:52.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:52 smithi002 ceph-mon[31328]: pgmap v188: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:53.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 475471 8.49 MB/sec execute 173 sec latency 45.306 ms 2024-03-11T18:39:54.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:54 smithi165 ceph-mon[36634]: pgmap v189: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:54.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:54 smithi023 ceph-mon[36406]: pgmap v189: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:54.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 476707 8.47 MB/sec execute 174 sec latency 18.447 ms 2024-03-11T18:39:54.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:54 smithi002 ceph-mon[31328]: pgmap v189: 129 pgs: 129 active+clean; 66 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-03-11T18:39:55.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 478068 8.46 MB/sec execute 175 sec latency 33.137 ms 2024-03-11T18:39:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:56 smithi023 ceph-mon[36406]: pgmap v190: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:56.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:56 smithi165 ceph-mon[36634]: pgmap v190: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:56.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 479937 8.49 MB/sec execute 176 sec latency 12.928 ms 2024-03-11T18:39:56.513 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:56 smithi002 ceph-mon[31328]: pgmap v190: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:57.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 480906 8.45 MB/sec execute 177 sec latency 32.740 ms 2024-03-11T18:39:58.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:39:58 smithi165 ceph-mon[36634]: pgmap v191: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:39:58 smithi023 ceph-mon[36406]: pgmap v191: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:58.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 483032 8.47 MB/sec execute 178 sec latency 44.300 ms 2024-03-11T18:39:58.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:39:58 smithi002 ceph-mon[31328]: pgmap v191: 129 pgs: 129 active+clean; 69 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-03-11T18:39:59.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 484041 8.46 MB/sec execute 179 sec latency 35.319 ms 2024-03-11T18:40:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:00 smithi165 ceph-mon[36634]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:00 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:40:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:00 smithi165 ceph-mon[36634]: overall HEALTH_OK 2024-03-11T18:40:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:00 smithi023 ceph-mon[36406]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:00 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:40:00.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:00 smithi023 ceph-mon[36406]: overall HEALTH_OK 2024-03-11T18:40:00.414 INFO:tasks.workunit.client.0.smithi002.stdout: 1 486450 8.48 MB/sec execute 180 sec latency 23.865 ms 2024-03-11T18:40:00.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:00 smithi002 ceph-mon[31328]: pgmap v192: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:00.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:00 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:40:00.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:00 smithi002 ceph-mon[31328]: overall HEALTH_OK 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 -- 172.21.15.2:0/3829983021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28073e40 msgr2=0x7f6e28111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/3829983021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28073e40 0x7f6e28111c00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009900 tx=0x7f6e1402bcc0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 -- 172.21.15.2:0/3829983021 shutdown_connections 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/3829983021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28112140 0x7f6e28114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/3829983021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28073e40 0x7f6e28111c00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/3829983021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 0x7f6e28073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 -- 172.21.15.2:0/3829983021 >> 172.21.15.2:0/3829983021 conn(0x7f6e280fcc00 msgr2=0x7f6e280ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 -- 172.21.15.2:0/3829983021 shutdown_connections 2024-03-11T18:40:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.139+0000 7f6e2fc00640 1 -- 172.21.15.2:0/3829983021 wait complete. 2024-03-11T18:40:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.140+0000 7f6e2fc00640 1 Processor -- start 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.140+0000 7f6e2fc00640 1 -- start start 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 0x7f6e2819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 0x7f6e2819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e281a2cd0 con 0x7f6e28073520 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e281a2e40 con 0x7f6e28073e40 2024-03-11T18:40:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2fc00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e281a3000 con 0x7f6e28112140 2024-03-11T18:40:01.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:45064/0 (socket says 172.21.15.2:45064) 2024-03-11T18:40:01.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2dc00640 1 -- 172.21.15.2:0/2829085821 learned_addr learned my addr 172.21.15.2:0/2829085821 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:01.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e2d200640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 0x7f6e2819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.141+0000 7f6e27e00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 0x7f6e2819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e2dc00640 1 -- 172.21.15.2:0/2829085821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 msgr2=0x7f6e2819e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 0x7f6e2819e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e2dc00640 1 -- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 msgr2=0x7f6e2819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 0x7f6e2819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e2dc00640 1 -- 172.21.15.2:0/2829085821 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009490 con 0x7f6e28112140 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.142+0000 7f6e27e00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 0x7f6e2819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.143+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009c90 tx=0x7f6e2000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.143+0000 7f6e25600640 1 -- 172.21.15.2:0/2829085821 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e20003b70 con 0x7f6e28112140 2024-03-11T18:40:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.143+0000 7f6e25600640 1 -- 172.21.15.2:0/2829085821 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e20003cd0 con 0x7f6e28112140 2024-03-11T18:40:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.143+0000 7f6e25600640 1 -- 172.21.15.2:0/2829085821 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e200108f0 con 0x7f6e28112140 2024-03-11T18:40:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.143+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e20009840 con 0x7f6e28112140 2024-03-11T18:40:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.144+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14009560 con 0x7f6e28112140 2024-03-11T18:40:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 Processor -- start 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e25600640 1 -- 172.21.15.2:0/2829085821 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f6e20004020 con 0x7f6e28112140 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 -- start start 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 0x55b1fbba6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbba7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbb62620 con 0x55b1fbba7b20 2024-03-11T18:40:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbb148c0 con 0x55b1fbba6db0 2024-03-11T18:40:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.148+0000 7f6e329d3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbb11df0 con 0x55b1fbbb0330 2024-03-11T18:40:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.149+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 0x55b1fbba6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.149+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 0x55b1fbba6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47838/0 (socket says 172.21.15.2:47838) 2024-03-11T18:40:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.149+0000 7f6e27e00640 1 -- 172.21.15.2:0/861440929 learned_addr learned my addr 172.21.15.2:0/861440929 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:01.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.149+0000 7f6e27e00640 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.149+0000 7f6e2d200640 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbba7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.153 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 -- 172.21.15.2:0/861440929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 msgr2=0x55b1fbba7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbba7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 -- 172.21.15.2:0/861440929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 msgr2=0x55b1fbba6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 0x55b1fbba6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 -- 172.21.15.2:0/861440929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1fba8eb00 con 0x55b1fbbb0330 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e25600640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 0x7f6e100a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e25600640 1 -- 172.21.15.2:0/2829085821 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e20014070 con 0x7f6e28112140 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e27e00640 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb2720 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e14000c00 tx=0x7f6e1402bcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96e654057b95806b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.150+0000 7f6e2d200640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 0x7f6e100a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.151+0000 7f6e1a000640 1 -- 172.21.15.2:0/861440929 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e140044d0 con 0x55b1fbbb0330 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.151+0000 7f6e1a000640 1 -- 172.21.15.2:0/861440929 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e1402cd50 con 0x55b1fbbb0330 2024-03-11T18:40:01.154 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.151+0000 7f6e1a000640 1 -- 172.21.15.2:0/861440929 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14005620 con 0x55b1fbbb0330 2024-03-11T18:40:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 -- 172.21.15.2:0/861440929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 msgr2=0x55b1fbbb2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e2d200640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 0x7f6e100a40d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b1fbba8280 tx=0x7f6e1c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb2720 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e14000c00 tx=0x7f6e1402bcc0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.155 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 -- 172.21.15.2:0/861440929 shutdown_connections 2024-03-11T18:40:01.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb2720 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbba7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 --2- 172.21.15.2:0/861440929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba7b20 0x55b1fbba6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 -- 172.21.15.2:0/861440929 >> 172.21.15.2:0/861440929 conn(0x55b1fbb6e4b0 msgr2=0x55b1fbba1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.152+0000 7f6e329d3740 1 -- 172.21.15.2:0/861440929 shutdown_connections 2024-03-11T18:40:01.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.153+0000 7f6e329d3740 1 -- 172.21.15.2:0/861440929 wait complete. 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.153+0000 7f6e329d3740 1 -- start start 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.153+0000 7f6e329d3740 1 Processor -- start 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbad910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbb0d5d0 con 0x55b1fbbb0330 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbab2b50 con 0x55b1fbba6db0 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbab27f0 con 0x55b1fbba7b20 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e329d3740 10 client.0 _ll_register_callbacks cb 0x55b1fbac6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e2d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e2d200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34026/0 (socket says 172.21.15.2:34026) 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.154+0000 7f6e2d200640 1 -- 172.21.15.2:0/328523172 learned_addr learned my addr 172.21.15.2:0/328523172 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e27e00640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 -- 172.21.15.2:0/328523172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 msgr2=0x55b1fbbad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 -- 172.21.15.2:0/328523172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 msgr2=0x55b1fbbade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 -- 172.21.15.2:0/328523172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e28114e60 con 0x55b1fbba6db0 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e2d200640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c01ea00 tx=0x7f6e1c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbeaca56976c2229 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.155+0000 7f6e0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.156+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c023070 con 0x55b1fbba6db0 2024-03-11T18:40:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.156+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e1c029420 con 0x55b1fbba6db0 2024-03-11T18:40:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.156+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b1fbadfcd0 con 0x55b1fbba6db0 2024-03-11T18:40:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.157+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e1c017070 con 0x55b1fbba6db0 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.157+0000 7f6e329d3740 10 client.25985 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.157+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b1fbb0bc50 con 0x55b1fbba6db0 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.157+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b1fbadfcd0 con 0x55b1fbba6db0 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.158+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6e1c031070 con 0x55b1fbba6db0 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.158+0000 7f6e329d3740 10 client.25985 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.158+0000 7f6e329d3740 10 client.25985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.158+0000 7f6e329d3740 20 client.25985 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.158+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbbc0370 0x55b1fbbc2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.159+0000 7f6e329d3740 4 client.25985 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:40:01.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.159+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b1fbb03ce0 con 0x55b1fbbc0370 2024-03-11T18:40:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.159+0000 7f6e27e00640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbbc0370 0x55b1fbbc2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.160+0000 7f6e27e00640 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbbc0370 0x55b1fbbc2760 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cc1c81f796d5d59 server_cookie=737bf54fd05e97d4 in_seq=0 out_seq=0 2024-03-11T18:40:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.160+0000 7f6e0ea00640 10 client.25985 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.160+0000 7f6e0ea00640 1 -- 172.21.15.2:0/328523172 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b1fbb03ce0 con 0x55b1fbbc0370 2024-03-11T18:40:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.160+0000 7f6e0ea00640 10 client.25985 handle_command_reply: tid=0 2024-03-11T18:40:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 client.25985 shutdown 2024-03-11T18:40:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 msgr2=0x55b1fbbad3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c01ea00 tx=0x7f6e1c004730 comp rx=0 tx=0).stop 2024-03-11T18:40:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 shutdown_connections 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbbc0370 0x55b1fbbc2760 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 --2- 172.21.15.2:0/328523172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbad3d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 >> 172.21.15.2:0/328523172 conn(0x55b1fbb6e4b0 msgr2=0x55b1fbba37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.161+0000 7f6e0ea00640 0 client.25985 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.162+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 shutdown_connections 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.162+0000 7f6e329d3740 1 -- 172.21.15.2:0/328523172 wait complete. 2024-03-11T18:40:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.162+0000 7f6e329d3740 20 client.25985 trim_cache size 0 max 16384 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 Processor -- start 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 -- start start 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba6db0 0x55b1fbbafb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fb855e60 con 0x55b1fbba6db0 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbadfcd0 con 0x55b1fbba7b20 2024-03-11T18:40:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.182+0000 7f6e329d3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbba8110 con 0x55b1fbbb0330 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e2dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e2dc00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:45086/0 (socket says 172.21.15.2:45086) 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34034/0 (socket says 172.21.15.2:34034) 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e2dc00640 1 -- 172.21.15.2:0/3637756525 learned_addr learned my addr 172.21.15.2:0/3637756525 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e2d200640 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba6db0 0x55b1fbbafb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 -- 172.21.15.2:0/3637756525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 msgr2=0x55b1fbbb7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 -- 172.21.15.2:0/3637756525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba6db0 msgr2=0x55b1fbbafb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba6db0 0x55b1fbbafb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e27e00640 1 -- 172.21.15.2:0/3637756525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b1fbaa8530 con 0x55b1fbba7b20 2024-03-11T18:40:01.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.183+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.184+0000 7f6e27e00640 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6e1402c2d0 tx=0x7f6e140045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae93c92c16b2929 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.184+0000 7f6e19600640 1 -- 172.21.15.2:0/3637756525 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1402c680 con 0x55b1fbba7b20 2024-03-11T18:40:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.184+0000 7f6e19600640 1 -- 172.21.15.2:0/3637756525 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e140052d0 con 0x55b1fbba7b20 2024-03-11T18:40:01.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.184+0000 7f6e19600640 1 -- 172.21.15.2:0/3637756525 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1404f320 con 0x55b1fbba7b20 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 -- 172.21.15.2:0/3637756525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 msgr2=0x55b1fbbb7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6e1402c2d0 tx=0x7f6e140045f0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 -- 172.21.15.2:0/3637756525 shutdown_connections 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb7120 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 --2- 172.21.15.2:0/3637756525 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbba6db0 0x55b1fbbafb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 -- 172.21.15.2:0/3637756525 >> 172.21.15.2:0/3637756525 conn(0x55b1fbb6e4b0 msgr2=0x55b1fbba15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 -- 172.21.15.2:0/3637756525 shutdown_connections 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.185+0000 7f6e329d3740 1 -- 172.21.15.2:0/3637756525 wait complete. 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.186+0000 7f6e329d3740 1 -- start start 2024-03-11T18:40:01.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.186+0000 7f6e329d3740 1 Processor -- start 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbb7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbbb4180 con 0x55b1fbbb0330 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbbb42f0 con 0x55b1fbba6db0 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1fbbb44b0 con 0x55b1fbba7b20 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e329d3740 10 client.0 _ll_register_callbacks cb 0x7f6e1c01e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:45094/0 (socket says 172.21.15.2:45094) 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 -- 172.21.15.2:0/251874411 learned_addr learned my addr 172.21.15.2:0/251874411 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:01.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e2d200640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbb7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 -- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 msgr2=0x55b1fbbb7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbb7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.187+0000 7f6e27e00640 1 -- 172.21.15.2:0/251874411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 msgr2=0x55b1fbbb3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e27e00640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e27e00640 1 -- 172.21.15.2:0/251874411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14009560 con 0x55b1fbba7b20 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e2d200640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbb7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e2dc00640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e27e00640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6e1402c200 tx=0x7f6e1402c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa73fbe09f97392 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14004ff0 con 0x55b1fbba7b20 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f6e14000e40 con 0x55b1fbba7b20 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.188+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b1fbb32d30 con 0x55b1fbba7b20 2024-03-11T18:40:01.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e14005150 con 0x55b1fbba7b20 2024-03-11T18:40:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e329d3740 10 client.25984 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b1fba8eb00 con 0x55b1fbba7b20 2024-03-11T18:40:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b1fbb32d30 con 0x55b1fbba7b20 2024-03-11T18:40:01.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6e14056070 con 0x55b1fbba7b20 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e329d3740 10 client.25984 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e329d3740 10 client.25984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.189+0000 7f6e329d3740 20 client.25984 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbc17900 0x55b1fbc19d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e329d3740 4 client.25984 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b1fb80d600 con 0x55b1fbc17900 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e2d200640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbc17900 0x55b1fbc19d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e2d200640 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbc17900 0x55b1fbc19d20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac8b4a4727488bb3 server_cookie=522290156dba0190 in_seq=0 out_seq=0 2024-03-11T18:40:01.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.190+0000 7f6e0ea00640 10 client.25984 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.406+0000 7f6e0ea00640 1 -- 172.21.15.2:0/251874411 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b1fb80d600 con 0x55b1fbc17900 2024-03-11T18:40:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.407+0000 7f6e0ea00640 10 client.25984 handle_command_reply: tid=0 2024-03-11T18:40:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.407+0000 7f6e329d3740 1 client.25984 shutdown 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.407+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 msgr2=0x55b1fbbb3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.407+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6e1402c200 tx=0x7f6e1402c700 comp rx=0 tx=0).stop 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 shutdown_connections 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b1fbc17900 0x55b1fbc19d20 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b1fbbb0330 0x55b1fbbb3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b1fbba7b20 0x55b1fbbb3700 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 --2- 172.21.15.2:0/251874411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b1fbba6db0 0x55b1fbbb7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 >> 172.21.15.2:0/251874411 conn(0x55b1fbb6e4b0 msgr2=0x55b1fbbabbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e0ea00640 0 client.25984 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 shutdown_connections 2024-03-11T18:40:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 1 -- 172.21.15.2:0/251874411 wait complete. 2024-03-11T18:40:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.408+0000 7f6e329d3740 20 client.25984 trim_cache size 0 max 16384 2024-03-11T18:40:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:40:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:40:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:40:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:40:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.410+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 msgr2=0x7f6e100a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.410+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 0x7f6e100a40d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b1fbba8280 tx=0x7f6e1c005f90 comp rx=0 tx=0).stop 2024-03-11T18:40:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.410+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 msgr2=0x7f6e281a26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.410+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009c90 tx=0x7f6e2000c6a0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 shutdown_connections 2024-03-11T18:40:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e28112140 0x7f6e281a26d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e28073e40 0x7f6e2819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e100a1c10 0x7f6e100a40d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 --2- 172.21.15.2:0/2829085821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e28073520 0x7f6e2819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 >> 172.21.15.2:0/2829085821 conn(0x7f6e280fcc00 msgr2=0x7f6e280fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 shutdown_connections 2024-03-11T18:40:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:01.411+0000 7f6e2fc00640 1 -- 172.21.15.2:0/2829085821 wait complete. 2024-03-11T18:40:01.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 486477 8.44 MB/sec execute 181 sec latency 199.838 ms 2024-03-11T18:40:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:40:01.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:40:01.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:01 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:01 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:40:01.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:01 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:40:02.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 487561 8.42 MB/sec execute 182 sec latency 223.479 ms 2024-03-11T18:40:02.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:02 smithi002 ceph-mon[31328]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-11T18:40:02.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:02 smithi165 ceph-mon[36634]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-11T18:40:02.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:02 smithi023 ceph-mon[36406]: pgmap v193: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.0 MiB/s wr, 164 op/s 2024-03-11T18:40:03.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 488286 8.40 MB/sec execute 183 sec latency 96.358 ms 2024-03-11T18:40:03.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:03 smithi002 ceph-mon[31328]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T18:40:03.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:03 smithi165 ceph-mon[36634]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T18:40:03.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:03 smithi023 ceph-mon[36406]: osdmap e98: 12 total, 12 up, 12 in 2024-03-11T18:40:04.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 489584 8.39 MB/sec execute 184 sec latency 122.201 ms 2024-03-11T18:40:04.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:04 smithi002 ceph-mon[31328]: pgmap v195: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:04.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:04 smithi165 ceph-mon[36634]: pgmap v195: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:04.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:04 smithi023 ceph-mon[36406]: pgmap v195: 129 pgs: 129 active+clean; 74 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-03-11T18:40:05.415 INFO:tasks.workunit.client.0.smithi002.stdout: 1 490867 8.38 MB/sec execute 185 sec latency 26.010 ms 2024-03-11T18:40:06.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 492711 8.38 MB/sec execute 186 sec latency 16.140 ms 2024-03-11T18:40:06.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:06 smithi002 ceph-mon[31328]: pgmap v196: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:06.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:06 smithi165 ceph-mon[36634]: pgmap v196: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:06.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:06 smithi023 ceph-mon[36406]: pgmap v196: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:07.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 494406 8.40 MB/sec execute 187 sec latency 16.332 ms 2024-03-11T18:40:08.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 496229 8.39 MB/sec execute 188 sec latency 16.539 ms 2024-03-11T18:40:08.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:08 smithi002 ceph-mon[31328]: pgmap v197: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:08.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:08 smithi165 ceph-mon[36634]: pgmap v197: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:08.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:08 smithi023 ceph-mon[36406]: pgmap v197: 129 pgs: 18 active+clean+snaptrim, 67 active+clean, 44 active+clean+snaptrim_wait; 67 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-11T18:40:09.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 497814 8.41 MB/sec execute 189 sec latency 102.163 ms 2024-03-11T18:40:10.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 498959 8.39 MB/sec execute 190 sec latency 25.689 ms 2024-03-11T18:40:10.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:10 smithi002 ceph-mon[31328]: pgmap v198: 129 pgs: 4 active+clean+snaptrim, 113 active+clean, 12 active+clean+snaptrim_wait; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.9 MiB/s wr, 327 op/s 2024-03-11T18:40:10.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:10 smithi165 ceph-mon[36634]: pgmap v198: 129 pgs: 4 active+clean+snaptrim, 113 active+clean, 12 active+clean+snaptrim_wait; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.9 MiB/s wr, 327 op/s 2024-03-11T18:40:10.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:10 smithi023 ceph-mon[36406]: pgmap v198: 129 pgs: 4 active+clean+snaptrim, 113 active+clean, 12 active+clean+snaptrim_wait; 70 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 1.9 MiB/s wr, 327 op/s 2024-03-11T18:40:11.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 501153 8.42 MB/sec execute 191 sec latency 12.414 ms 2024-03-11T18:40:12.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 501931 8.38 MB/sec execute 192 sec latency 38.434 ms 2024-03-11T18:40:12.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:12 smithi002 ceph-mon[31328]: pgmap v199: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-03-11T18:40:12.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:12 smithi165 ceph-mon[36634]: pgmap v199: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-03-11T18:40:12.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:12 smithi023 ceph-mon[36406]: pgmap v199: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.0 MiB/s wr, 348 op/s 2024-03-11T18:40:13.416 INFO:tasks.workunit.client.0.smithi002.stdout: 1 504136 8.40 MB/sec execute 193 sec latency 12.313 ms 2024-03-11T18:40:14.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 505043 8.39 MB/sec execute 194 sec latency 123.485 ms 2024-03-11T18:40:14.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:14 smithi002 ceph-mon[31328]: pgmap v200: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.9 MiB/s wr, 336 op/s 2024-03-11T18:40:14.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:14 smithi002 ceph-mon[31328]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T18:40:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.561+0000 7f04d9200640 1 -- 172.21.15.2:0/3561199153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 msgr2=0x7f04d41015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.561+0000 7f04d9200640 1 --2- 172.21.15.2:0/3561199153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d41015a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009890 tx=0x7f04c402bb70 comp rx=0 tx=0).stop 2024-03-11T18:40:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 -- 172.21.15.2:0/3561199153 shutdown_connections 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 --2- 172.21.15.2:0/3561199153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4106290 0x7f04d410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 --2- 172.21.15.2:0/3561199153 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4101b70 0x7f04d4105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 --2- 172.21.15.2:0/3561199153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d41015a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 -- 172.21.15.2:0/3561199153 >> 172.21.15.2:0/3561199153 conn(0x7f04d40fca80 msgr2=0x7f04d40feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 -- 172.21.15.2:0/3561199153 shutdown_connections 2024-03-11T18:40:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.562+0000 7f04d9200640 1 -- 172.21.15.2:0/3561199153 wait complete. 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.563+0000 7f04d9200640 1 Processor -- start 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.563+0000 7f04d9200640 1 -- start start 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4199d20 0x7f04d419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a710 con 0x7f04d4199d20 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a880 con 0x7f04d41011c0 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d9200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d419a9f0 con 0x7f04d4106290 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57322/0 (socket says 172.21.15.2:57322) 2024-03-11T18:40:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46150/0 (socket says 172.21.15.2:46150) 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 -- 172.21.15.2:0/769662122 learned_addr learned my addr 172.21.15.2:0/769662122 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04d3e00640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4199d20 0x7f04d419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 -- 172.21.15.2:0/769662122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 msgr2=0x7f04d4195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 -- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4199d20 msgr2=0x7f04d419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4199d20 0x7f04d419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.564+0000 7f04cbe00640 1 -- 172.21.15.2:0/769662122 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4009490 con 0x7f04d4106290 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d3400640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04cbe00640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00e9f0 tx=0x7f04bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d0c00640 1 -- 172.21.15.2:0/769662122 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc009600 con 0x7f04d4106290 2024-03-11T18:40:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d0c00640 1 -- 172.21.15.2:0/769662122 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04bc00ccc0 con 0x7f04d4106290 2024-03-11T18:40:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d0c00640 1 -- 172.21.15.2:0/769662122 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc018880 con 0x7f04d4106290 2024-03-11T18:40:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d41a1fd0 con 0x7f04d4106290 2024-03-11T18:40:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.565+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d41a23a0 con 0x7f04d4106290 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.570+0000 7f04dd5a7740 1 Processor -- start 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.570+0000 7f04d0c00640 1 -- 172.21.15.2:0/769662122 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f04bc010a80 con 0x7f04d4106290 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.570+0000 7f04dd5a7740 1 -- start start 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1ceff0 0x55d6de1cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 0x55d6de1da960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de18a620 con 0x55d6de1d8570 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de13c8c0 con 0x55d6de1cfd60 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de139df0 con 0x55d6de1ceff0 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46166/0 (socket says 172.21.15.2:46166) 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 -- 172.21.15.2:0/3676119829 learned_addr learned my addr 172.21.15.2:0/3676119829 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3400640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1ceff0 0x55d6de1cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04cbe00640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 0x55d6de1da960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 -- 172.21.15.2:0/3676119829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1ceff0 msgr2=0x55d6de1cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1ceff0 0x55d6de1cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 -- 172.21.15.2:0/3676119829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 msgr2=0x55d6de1da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 0x55d6de1da960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04d3e00640 1 -- 172.21.15.2:0/3676119829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6de0b6b00 con 0x55d6de1cfd60 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.571+0000 7f04cbe00640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 0x55d6de1da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:40:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04d3e00640 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009f90 tx=0x7f04cc010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38d687e6b58f3b60 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04d0c00640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 0x7f04b80a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04c8c00640 1 -- 172.21.15.2:0/3676119829 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc011700 con 0x55d6de1cfd60 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04c8c00640 1 -- 172.21.15.2:0/3676119829 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04cc015070 con 0x55d6de1cfd60 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04c8c00640 1 -- 172.21.15.2:0/3676119829 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc019cd0 con 0x55d6de1cfd60 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04d0c00640 1 -- 172.21.15.2:0/769662122 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f04bc014070 con 0x7f04d4106290 2024-03-11T18:40:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.572+0000 7f04d3400640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 0x7f04b80a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- 172.21.15.2:0/3676119829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 msgr2=0x55d6de1ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009f90 tx=0x7f04cc010860 comp rx=0 tx=0).stop 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- 172.21.15.2:0/3676119829 shutdown_connections 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1d8570 0x55d6de1da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1ceff0 0x55d6de1cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/3676119829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1ceab0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- 172.21.15.2:0/3676119829 >> 172.21.15.2:0/3676119829 conn(0x55d6de1964b0 msgr2=0x55d6de1c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- 172.21.15.2:0/3676119829 shutdown_connections 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04d3400640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 0x7f04b80a8450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04c402c180 tx=0x7f04c402e040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- 172.21.15.2:0/3676119829 wait complete. 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.573+0000 7f04dd5a7740 1 -- start start 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 Processor -- start 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1cfd60 0x55d6de1d5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de1355d0 con 0x55d6de1ceff0 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de0dab50 con 0x55d6de1d8570 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de0da7f0 con 0x55d6de1cfd60 2024-03-11T18:40:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.574+0000 7f04dd5a7740 10 client.0 _ll_register_callbacks cb 0x55d6de0ee550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46170/0 (socket says 172.21.15.2:46170) 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 -- 172.21.15.2:0/2760511888 learned_addr learned my addr 172.21.15.2:0/2760511888 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04cbe00640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1cfd60 0x55d6de1d5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 -- 172.21.15.2:0/2760511888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1cfd60 msgr2=0x55d6de1d5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1cfd60 0x55d6de1d5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 -- 172.21.15.2:0/2760511888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 msgr2=0x55d6de1d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 -- 172.21.15.2:0/2760511888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04cc009bf0 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f04cc010d60 tx=0x7f04cc0195b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58bfd7bc52db339f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.575+0000 7f04c1600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc027070 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f04cc004a40 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6de107cd0 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04cc004c00 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04dd5a7740 10 client.26003 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.576+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d6de133c50 con 0x55d6de1d8570 2024-03-11T18:40:14.583 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d6de107cd0 con 0x55d6de1d8570 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f04cc02a020 con 0x55d6de1d8570 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 10 client.26003 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 10 client.26003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 20 client.26003 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1e84a0 0x55d6de1ea890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 4 client.26003 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6de12bce0 con 0x55d6de1e84a0 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.577+0000 7f04d3400640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1e84a0 0x55d6de1ea890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.578+0000 7f04d3400640 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1e84a0 0x55d6de1ea890 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f137ce1a489cecba server_cookie=a6f8364c478b4e6d in_seq=0 out_seq=0 2024-03-11T18:40:14.584 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.578+0000 7f04c1600640 10 client.26003 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.578+0000 7f04c1600640 1 -- 172.21.15.2:0/2760511888 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d6de12bce0 con 0x55d6de1e84a0 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.578+0000 7f04c1600640 10 client.26003 handle_command_reply: tid=0 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 client.26003 shutdown 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 msgr2=0x55d6de1d6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f04cc010d60 tx=0x7f04cc0195b0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 shutdown_connections 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1e84a0 0x55d6de1ea890 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1d8570 0x55d6de1d6070 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1cfd60 0x55d6de1d5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04c1600640 0 client.26003 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2760511888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 >> 172.21.15.2:0/2760511888 conn(0x55d6de1964b0 msgr2=0x55d6de1cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 shutdown_connections 2024-03-11T18:40:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2760511888 wait complete. 2024-03-11T18:40:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.579+0000 7f04dd5a7740 20 client.26003 trim_cache size 0 max 16384 2024-03-11T18:40:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.594+0000 7f04dd5a7740 1 Processor -- start 2024-03-11T18:40:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 -- start start 2024-03-11T18:40:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1df390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1df8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6dde7de60 con 0x55d6de1ceff0 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de1c55a0 con 0x55d6de1cfd60 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de1e5110 con 0x55d6de1964b0 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04d3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04d3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57360/0 (socket says 172.21.15.2:57360) 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04d3400640 1 -- 172.21.15.2:0/2767340879 learned_addr learned my addr 172.21.15.2:0/2767340879 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.595+0000 7f04d3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1df8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04cbe00640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1df390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 -- 172.21.15.2:0/2767340879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 msgr2=0x55d6de1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 -- 172.21.15.2:0/2767340879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 msgr2=0x55d6de1df390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1df390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 -- 172.21.15.2:0/2767340879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6de0d0530 con 0x55d6de1cfd60 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04cbe00640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1df390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3400640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.596+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1df8d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009850 tx=0x7f04cc005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7e7db44ce421f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.597+0000 7f04c3e00640 1 -- 172.21.15.2:0/2767340879 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc027070 con 0x55d6de1cfd60 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.597+0000 7f04c3e00640 1 -- 172.21.15.2:0/2767340879 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04cc015030 con 0x55d6de1cfd60 2024-03-11T18:40:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.597+0000 7f04c3e00640 1 -- 172.21.15.2:0/2767340879 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc019930 con 0x55d6de1cfd60 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2767340879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 msgr2=0x55d6de1df8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1df8d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009850 tx=0x7f04cc005bd0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2767340879 shutdown_connections 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1df8d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1df390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2767340879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2767340879 >> 172.21.15.2:0/2767340879 conn(0x55d6de1dab40 msgr2=0x55d6de1d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2767340879 shutdown_connections 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2767340879 wait complete. 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.598+0000 7f04dd5a7740 1 -- start start 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 Processor -- start 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de23e940 con 0x55d6de1ceff0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de23ea80 con 0x55d6de1cfd60 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6de23ebf0 con 0x55d6de1964b0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04dd5a7740 10 client.0 _ll_register_callbacks cb 0x7f04bc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04d3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.599+0000 7f04cbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:57374/0 (socket says 172.21.15.2:57374) 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 -- 172.21.15.2:0/2552548282 learned_addr learned my addr 172.21.15.2:0/2552548282 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 -- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 msgr2=0x55d6de1d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 -- 172.21.15.2:0/2552548282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 msgr2=0x55d6de1d2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 -- 172.21.15.2:0/2552548282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d410e670 con 0x55d6de1964b0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3e00640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04d3400640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f04c4037040 tx=0x7f04c402bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cae99ddc3368a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.600+0000 7f04c1600640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c402c710 con 0x55d6de1964b0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f04c402c870 con 0x55d6de1964b0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6de15ad30 con 0x55d6de1964b0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04c402ca30 con 0x55d6de1964b0 2024-03-11T18:40:14.606 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04dd5a7740 10 client.26002 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.601+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d6de133c50 con 0x55d6de1964b0 2024-03-11T18:40:14.607 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d6de15ad30 con 0x55d6de1964b0 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f04c4047040 con 0x55d6de1964b0 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 10 client.26002 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 10 client.26002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 20 client.26002 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1d8570 0x55d6de2424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 4 client.26002 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d6dde35600 con 0x55d6de1d8570 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.602+0000 7f04cbe00640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1d8570 0x55d6de2424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.603+0000 7f04cbe00640 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1d8570 0x55d6de2424c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffdf164d9eea3525 server_cookie=de3d8adc3bdc6db1 in_seq=0 out_seq=0 2024-03-11T18:40:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.603+0000 7f04c1600640 10 client.26002 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:14.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:14 smithi165 ceph-mon[36634]: pgmap v200: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.9 MiB/s wr, 336 op/s 2024-03-11T18:40:14.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:14 smithi165 ceph-mon[36634]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T18:40:14.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:14 smithi023 ceph-mon[36406]: pgmap v200: 129 pgs: 129 active+clean; 68 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 1.9 MiB/s wr, 336 op/s 2024-03-11T18:40:14.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:14 smithi023 ceph-mon[36406]: osdmap e99: 12 total, 12 up, 12 in 2024-03-11T18:40:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04c1600640 1 -- 172.21.15.2:0/2552548282 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d6dde35600 con 0x55d6de1d8570 2024-03-11T18:40:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04c1600640 10 client.26002 handle_command_reply: tid=0 2024-03-11T18:40:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 client.26002 shutdown 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 msgr2=0x55d6de1d22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f04c4037040 tx=0x7f04c402bb70 comp rx=0 tx=0).stop 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 shutdown_connections 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d6de1d8570 0x55d6de2424c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6de1cfd60 0x55d6de1d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d6de1ceff0 0x55d6de1d2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 --2- 172.21.15.2:0/2552548282 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d6de1964b0 0x55d6de1d22c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04c1600640 0 client.26002 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 >> 172.21.15.2:0/2552548282 conn(0x55d6de1dab40 msgr2=0x55d6de1cb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.824 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 shutdown_connections 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.820+0000 7f04dd5a7740 1 -- 172.21.15.2:0/2552548282 wait complete. 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04dd5a7740 20 client.26002 trim_cache size 0 max 16384 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:40:14.825 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:40:14.826 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 msgr2=0x7f04b80a8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 0x7f04b80a8450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f04c402c180 tx=0x7f04c402e040 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 msgr2=0x7f04d41959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04bc00e9f0 tx=0x7f04bc00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 shutdown_connections 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f04b80a5f90 0x7f04b80a8450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04d4199d20 0x7f04d419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f04d4106290 0x7f04d41959a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 --2- 172.21.15.2:0/769662122 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04d41011c0 0x7f04d4195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:14.827 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.821+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 >> 172.21.15.2:0/769662122 conn(0x7f04d40fca80 msgr2=0x7f04d40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.822+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 shutdown_connections 2024-03-11T18:40:14.828 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:14.822+0000 7f04d9200640 1 -- 172.21.15.2:0/769662122 wait complete. 2024-03-11T18:40:15.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 506675 8.38 MB/sec execute 195 sec latency 151.240 ms 2024-03-11T18:40:16.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 508492 8.40 MB/sec execute 196 sec latency 13.299 ms 2024-03-11T18:40:16.469 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:16 smithi165 ceph-mon[36634]: pgmap v202: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:16.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:16 smithi002 ceph-mon[31328]: pgmap v202: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:16.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:16 smithi023 ceph-mon[36406]: pgmap v202: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:17.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 510239 8.39 MB/sec execute 197 sec latency 25.314 ms 2024-03-11T18:40:18.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 512044 8.41 MB/sec execute 198 sec latency 9.479 ms 2024-03-11T18:40:18.508 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:18 smithi002 ceph-mon[31328]: pgmap v203: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:18.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:18 smithi165 ceph-mon[36634]: pgmap v203: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:18.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:18 smithi023 ceph-mon[36406]: pgmap v203: 129 pgs: 129 active+clean; 75 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 2.5 MiB/s wr, 199 op/s 2024-03-11T18:40:19.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 513272 8.40 MB/sec execute 199 sec latency 89.352 ms 2024-03-11T18:40:20.417 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515516 8.42 MB/sec execute 200 sec latency 10.393 ms 2024-03-11T18:40:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:20 smithi002 ceph-mon[31328]: pgmap v204: 129 pgs: 129 active+clean; 82 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-03-11T18:40:20.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:20 smithi165 ceph-mon[36634]: pgmap v204: 129 pgs: 129 active+clean; 82 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-03-11T18:40:20.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:20 smithi023 ceph-mon[36406]: pgmap v204: 129 pgs: 129 active+clean; 82 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-03-11T18:40:21.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.38 MB/sec execute 201 sec latency 795.641 ms 2024-03-11T18:40:22.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.34 MB/sec execute 202 sec latency 1795.819 ms 2024-03-11T18:40:22.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:22 smithi002 ceph-mon[31328]: pgmap v205: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:22.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:22 smithi165 ceph-mon[36634]: pgmap v205: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:22.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:22 smithi023 ceph-mon[36406]: pgmap v205: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:23.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.30 MB/sec execute 203 sec latency 2795.919 ms 2024-03-11T18:40:24.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.26 MB/sec execute 204 sec latency 3796.066 ms 2024-03-11T18:40:24.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:24 smithi002 ceph-mon[31328]: pgmap v206: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:24.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:24 smithi165 ceph-mon[36634]: pgmap v206: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:24.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:24 smithi023 ceph-mon[36406]: pgmap v206: 129 pgs: 129 active+clean; 87 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 326 op/s 2024-03-11T18:40:25.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.22 MB/sec execute 205 sec latency 4796.225 ms 2024-03-11T18:40:26.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.18 MB/sec execute 206 sec latency 5796.362 ms 2024-03-11T18:40:26.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:26 smithi002 ceph-mon[31328]: pgmap v207: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-11T18:40:26.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:26 smithi165 ceph-mon[36634]: pgmap v207: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-11T18:40:26.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:26 smithi023 ceph-mon[36406]: pgmap v207: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 284 op/s 2024-03-11T18:40:27.418 INFO:tasks.workunit.client.0.smithi002.stdout: 1 515718 8.14 MB/sec execute 207 sec latency 6796.508 ms 2024-03-11T18:40:27.973 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 -- 172.21.15.2:0/3146907751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f34103f70 msgr2=0x7f7f34107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 --2- 172.21.15.2:0/3146907751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f34103f70 0x7f7f34107fb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7f20009900 tx=0x7f7f2002bc30 comp rx=0 tx=0).stop 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 -- 172.21.15.2:0/3146907751 shutdown_connections 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 --2- 172.21.15.2:0/3146907751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f34110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 --2- 172.21.15.2:0/3146907751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f34103f70 0x7f7f34107fb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 --2- 172.21.15.2:0/3146907751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f341035c0 0x7f7f341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.974 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 -- 172.21.15.2:0/3146907751 >> 172.21.15.2:0/3146907751 conn(0x7f7f340fee00 msgr2=0x7f7f34101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 -- 172.21.15.2:0/3146907751 shutdown_connections 2024-03-11T18:40:27.975 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.971+0000 7f7f3be00640 1 -- 172.21.15.2:0/3146907751 wait complete. 2024-03-11T18:40:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.972+0000 7f7f3be00640 1 Processor -- start 2024-03-11T18:40:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.972+0000 7f7f3be00640 1 -- start start 2024-03-11T18:40:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f341035c0 0x7f7f3419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f34199860 0x7f7f34199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3419a310 con 0x7f7f34199860 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3419a480 con 0x7f7f34108690 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f3be00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3419a5f0 con 0x7f7f341035c0 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f38a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f38a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38498/0 (socket says 172.21.15.2:38498) 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f38a00640 1 -- 172.21.15.2:0/1671434991 learned_addr learned my addr 172.21.15.2:0/1671434991 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f34199860 0x7f7f34199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f39400640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f341035c0 0x7f7f3419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.973+0000 7f7f38a00640 1 -- 172.21.15.2:0/1671434991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f341035c0 msgr2=0x7f7f3419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f341035c0 0x7f7f3419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f38a00640 1 -- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f34199860 msgr2=0x7f7f34199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f34199860 0x7f7f34199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f38a00640 1 -- 172.21.15.2:0/1671434991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20009560 con 0x7f7f34108690 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7f20000c00 tx=0x7f7f2002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f32000640 1 -- 172.21.15.2:0/1671434991 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2002dea0 con 0x7f7f34108690 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f32000640 1 -- 172.21.15.2:0/1671434991 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f200042d0 con 0x7f7f34108690 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f341a64f0 con 0x7f7f34108690 2024-03-11T18:40:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f32000640 1 -- 172.21.15.2:0/1671434991 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20003b50 con 0x7f7f34108690 2024-03-11T18:40:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.974+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f341a69d0 con 0x7f7f34108690 2024-03-11T18:40:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.978+0000 7f7f32000640 1 -- 172.21.15.2:0/1671434991 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f7f20003cb0 con 0x7f7f34108690 2024-03-11T18:40:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.978+0000 7f7f3ea3f740 1 Processor -- start 2024-03-11T18:40:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.978+0000 7f7f3ea3f740 1 -- start start 2024-03-11T18:40:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa585d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5868720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa5818620 con 0x556fa585cdb0 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa57ca8c0 con 0x556fa585db20 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa57c7df0 con 0x556fa5866330 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38504/0 (socket says 172.21.15.2:38504) 2024-03-11T18:40:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 -- 172.21.15.2:0/1055591884 learned_addr learned my addr 172.21.15.2:0/1055591884 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5868720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39400640 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa585d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 -- 172.21.15.2:0/1055591884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 msgr2=0x556fa5868720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5868720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 -- 172.21.15.2:0/1055591884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 msgr2=0x556fa585d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa585d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.979+0000 7f7f39e00640 1 -- 172.21.15.2:0/1055591884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fa5744b00 con 0x556fa585db20 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009f50 tx=0x7f7f2c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd313a33611c0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f32000640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 0x7f7f1c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f26600640 1 -- 172.21.15.2:0/1055591884 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004d20 con 0x556fa585db20 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f26600640 1 -- 172.21.15.2:0/1055591884 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c004e80 con 0x556fa585db20 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f26600640 1 -- 172.21.15.2:0/1055591884 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c005280 con 0x556fa585db20 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f32000640 1 -- 172.21.15.2:0/1671434991 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f7f200f0090 con 0x7f7f34108690 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.980+0000 7f7f39400640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 0x7f7f1c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1055591884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 msgr2=0x556fa585c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009f50 tx=0x7f7f2c0118d0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1055591884 shutdown_connections 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5868720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa585d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1055591884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585db20 0x556fa585c870 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1055591884 >> 172.21.15.2:0/1055591884 conn(0x556fa58244b0 msgr2=0x556fa5857aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1055591884 shutdown_connections 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f39400640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 0x7f7f1c0a4050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x556fa585e280 tx=0x7f7f28005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:27.987 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.981+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1055591884 wait complete. 2024-03-11T18:40:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.982+0000 7f7f3ea3f740 1 -- start start 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.982+0000 7f7f3ea3f740 1 Processor -- start 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa5863900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5863e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa57c35d0 con 0x556fa585db20 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa5768b50 con 0x556fa585cdb0 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa57687f0 con 0x556fa5866330 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f3ea3f740 10 client.0 _ll_register_callbacks cb 0x556fa577c550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f39400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5863e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f39400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38506/0 (socket says 172.21.15.2:38506) 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f38a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa5863900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.983+0000 7f7f39400640 1 -- 172.21.15.2:0/1703841352 learned_addr learned my addr 172.21.15.2:0/1703841352 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 -- 172.21.15.2:0/1703841352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 msgr2=0x556fa5863e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5863e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 -- 172.21.15.2:0/1703841352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 msgr2=0x556fa5863900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa5863900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 -- 172.21.15.2:0/1703841352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c009c00 con 0x556fa585cdb0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f39400640 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009560 tx=0x7f7f28004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece777d86d403339 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f1aa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28017070 con 0x556fa585cdb0 2024-03-11T18:40:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f7f2801e470 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.984+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556fa5795cd0 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f2801e630 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 10 client.26021 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556fa57c1c50 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556fa5795cd0 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7f28029a10 con 0x556fa585cdb0 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 10 client.26021 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 10 client.26021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 20 client.26021 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58764a0 0x556fa58788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 4 client.26021 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:40:27.995 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fa57b9ce0 con 0x556fa58764a0 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.985+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58764a0 0x556fa58788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.986+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58764a0 0x556fa58788c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=791c2a6febdd58d9 server_cookie=b183cf81c34d4b8 in_seq=0 out_seq=0 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.986+0000 7f7f1aa00640 10 client.26021 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.986+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1703841352 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556fa57b9ce0 con 0x556fa58764a0 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.986+0000 7f7f1aa00640 10 client.26021 handle_command_reply: tid=0 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 client.26021 shutdown 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 msgr2=0x556fa58633c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009560 tx=0x7f7f28004730 comp rx=0 tx=0).stop 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 shutdown_connections 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58764a0 0x556fa58788c0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa5866330 0x556fa5863e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.996 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa5863900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f1aa00640 0 client.26021 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1703841352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa585cdb0 0x556fa58633c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 >> 172.21.15.2:0/1703841352 conn(0x556fa58244b0 msgr2=0x556fa58597f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 shutdown_connections 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1703841352 wait complete. 2024-03-11T18:40:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:27.987+0000 7f7f3ea3f740 20 client.26021 trim_cache size 0 max 16384 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.002+0000 7f7f3ea3f740 1 Processor -- start 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 -- start start 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 0x556fa5863c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa5860640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa550be60 con 0x556fa585cdb0 2024-03-11T18:40:28.006 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa5858aa0 con 0x556fa5866330 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa587db10 con 0x556fa585db20 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f39400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa5860640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f39400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57244/0 (socket says 172.21.15.2:57244) 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f39400640 1 -- 172.21.15.2:0/2509778134 learned_addr learned my addr 172.21.15.2:0/2509778134 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.003+0000 7f7f38a00640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 0x556fa5863c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 -- 172.21.15.2:0/2509778134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 msgr2=0x556fa5863c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 0x556fa5863c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 -- 172.21.15.2:0/2509778134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 msgr2=0x556fa5865af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 -- 172.21.15.2:0/2509778134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fa575e530 con 0x556fa5866330 2024-03-11T18:40:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39400640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f38a00640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 0x556fa5863c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f39e00640 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa5860640 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c011dd0 tx=0x7f7f2c003a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8905afeb84e9bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f25c00640 1 -- 172.21.15.2:0/2509778134 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c004050 con 0x556fa5866330 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f25c00640 1 -- 172.21.15.2:0/2509778134 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f2c026070 con 0x556fa5866330 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.004+0000 7f7f25c00640 1 -- 172.21.15.2:0/2509778134 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c005260 con 0x556fa5866330 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/2509778134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 msgr2=0x556fa5860640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa5860640 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c011dd0 tx=0x7f7f2c003a40 comp rx=0 tx=0).stop 2024-03-11T18:40:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/2509778134 shutdown_connections 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa5860640 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585db20 0x556fa5863c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/2509778134 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585cdb0 0x556fa5865af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/2509778134 >> 172.21.15.2:0/2509778134 conn(0x556fa58244b0 msgr2=0x556fa5861c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.005+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/2509778134 shutdown_connections 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.006+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/2509778134 wait complete. 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.006+0000 7f7f3ea3f740 1 -- start start 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.006+0000 7f7f3ea3f740 1 Processor -- start 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 0x556fa586d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa586b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa586c320 con 0x556fa585db20 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa586c490 con 0x556fa5866330 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fa586c650 con 0x556fa585cdb0 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f3ea3f740 10 client.0 _ll_register_callbacks cb 0x7f7f28009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 0x556fa586d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f38a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa586b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:38524/0 (socket says 172.21.15.2:38524) 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 -- 172.21.15.2:0/1384627 learned_addr learned my addr 172.21.15.2:0/1384627 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 -- 172.21.15.2:0/1384627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 msgr2=0x556fa586d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:28.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 0x556fa586d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 -- 172.21.15.2:0/1384627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 msgr2=0x556fa586b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa586b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39e00640 1 -- 172.21.15.2:0/1384627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fa57c1c50 con 0x556fa5866330 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f39400640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 0x556fa586d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.007+0000 7f7f38a00640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa586b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f39e00640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c011910 tx=0x7f7f2c004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d0d40aa0df6402 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f1aa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0044d0 con 0x556fa5866330 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556fa57e8d30 con 0x556fa5866330 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f7f2c005000 con 0x556fa5866330 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f2c0051c0 con 0x556fa5866330 2024-03-11T18:40:28.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f3ea3f740 10 client.26027 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.008+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f7f341a69d0 con 0x556fa5866330 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556fa57e8d30 con 0x556fa5866330 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7f2c02a030 con 0x556fa5866330 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 10 client.26027 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 10 client.26027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 20 client.26027 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58d6550 0x556fa58d8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 4 client.26027 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556fa54c3600 con 0x556fa58d6550 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.009+0000 7f7f39400640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58d6550 0x556fa58d8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.010+0000 7f7f39400640 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58d6550 0x556fa58d8940 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3dbc8581539d5e server_cookie=fe883df9cb1caf7d in_seq=0 out_seq=0 2024-03-11T18:40:28.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.010+0000 7f7f1aa00640 10 client.26027 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f1aa00640 1 -- 172.21.15.2:0/1384627 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556fa54c3600 con 0x556fa58d6550 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f1aa00640 10 client.26027 handle_command_reply: tid=0 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 client.26027 shutdown 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 msgr2=0x556fa586bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c011910 tx=0x7f7f2c004990 comp rx=0 tx=0).stop 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 shutdown_connections 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x556fa58d6550 0x556fa58d8940 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556fa5866330 0x556fa586bce0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x556fa585db20 0x556fa586b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 --2- 172.21.15.2:0/1384627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x556fa585cdb0 0x556fa586d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f1aa00640 0 client.26027 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.137+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 >> 172.21.15.2:0/1384627 conn(0x556fa58244b0 msgr2=0x556fa58564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.138+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 shutdown_connections 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.138+0000 7f7f3ea3f740 1 -- 172.21.15.2:0/1384627 wait complete. 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.138+0000 7f7f3ea3f740 20 client.26027 trim_cache size 0 max 16384 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:40:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 msgr2=0x7f7f1c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 0x7f7f1c0a4050 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x556fa585e280 tx=0x7f7f28005f90 comp rx=0 tx=0).stop 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 msgr2=0x7f7f3419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7f20000c00 tx=0x7f7f2002c670 comp rx=0 tx=0).stop 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 shutdown_connections 2024-03-11T18:40:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7f1c0a1b90 0x7f7f1c0a4050 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7f34199860 0x7f7f34199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f34108690 0x7f7f3419fbe0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 --2- 172.21.15.2:0/1671434991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f341035c0 0x7f7f3419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 >> 172.21.15.2:0/1671434991 conn(0x7f7f340fee00 msgr2=0x7f7f34101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 shutdown_connections 2024-03-11T18:40:28.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:28.139+0000 7f7f3be00640 1 -- 172.21.15.2:0/1671434991 wait complete. 2024-03-11T18:40:28.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 516144 8.11 MB/sec execute 208 sec latency 7335.067 ms 2024-03-11T18:40:28.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:28 smithi002 ceph-mon[31328]: pgmap v208: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 239 op/s 2024-03-11T18:40:28.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:28 smithi165 ceph-mon[36634]: pgmap v208: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 239 op/s 2024-03-11T18:40:28.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:28 smithi023 ceph-mon[36406]: pgmap v208: 129 pgs: 129 active+clean; 88 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 239 op/s 2024-03-11T18:40:29.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 517408 8.09 MB/sec execute 209 sec latency 109.448 ms 2024-03-11T18:40:30.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 518891 8.11 MB/sec execute 210 sec latency 15.828 ms 2024-03-11T18:40:30.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:30 smithi023 ceph-mon[36406]: pgmap v209: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 308 op/s 2024-03-11T18:40:30.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:30 smithi165 ceph-mon[36634]: pgmap v209: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 308 op/s 2024-03-11T18:40:30.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:30 smithi002 ceph-mon[31328]: pgmap v209: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 308 op/s 2024-03-11T18:40:31.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 519617 8.08 MB/sec execute 211 sec latency 25.245 ms 2024-03-11T18:40:32.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 521009 8.07 MB/sec execute 212 sec latency 27.261 ms 2024-03-11T18:40:32.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:32 smithi023 ceph-mon[36406]: pgmap v210: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-11T18:40:32.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:32 smithi165 ceph-mon[36634]: pgmap v210: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-11T18:40:32.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:32 smithi002 ceph-mon[31328]: pgmap v210: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 2.0 MiB/s wr, 177 op/s 2024-03-11T18:40:33.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 522464 8.09 MB/sec execute 213 sec latency 16.147 ms 2024-03-11T18:40:34.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 523127 8.06 MB/sec execute 214 sec latency 85.012 ms 2024-03-11T18:40:34.554 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:34 smithi002 ceph-mon[31328]: pgmap v211: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-11T18:40:34.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:34 smithi165 ceph-mon[36634]: pgmap v211: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-11T18:40:34.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:34 smithi023 ceph-mon[36406]: pgmap v211: 129 pgs: 129 active+clean; 85 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-11T18:40:35.419 INFO:tasks.workunit.client.0.smithi002.stdout: 1 524683 8.05 MB/sec execute 215 sec latency 24.679 ms 2024-03-11T18:40:36.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 526366 8.07 MB/sec execute 216 sec latency 25.442 ms 2024-03-11T18:40:36.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:36 smithi165 ceph-mon[36634]: pgmap v212: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-03-11T18:40:36.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:36 smithi023 ceph-mon[36406]: pgmap v212: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-03-11T18:40:36.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:36 smithi002 ceph-mon[31328]: pgmap v212: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 2.3 MiB/s wr, 211 op/s 2024-03-11T18:40:37.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 527468 8.05 MB/sec execute 217 sec latency 24.286 ms 2024-03-11T18:40:38.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 529550 8.08 MB/sec execute 218 sec latency 20.192 ms 2024-03-11T18:40:38.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:38 smithi165 ceph-mon[36634]: pgmap v213: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-03-11T18:40:38.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:38 smithi023 ceph-mon[36406]: pgmap v213: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-03-11T18:40:38.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:38 smithi002 ceph-mon[31328]: pgmap v213: 129 pgs: 129 active+clean; 88 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 1.6 MiB/s wr, 158 op/s 2024-03-11T18:40:39.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 530250 8.05 MB/sec execute 219 sec latency 128.906 ms 2024-03-11T18:40:40.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 531783 8.04 MB/sec execute 220 sec latency 36.030 ms 2024-03-11T18:40:40.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:40 smithi165 ceph-mon[36634]: pgmap v214: 129 pgs: 129 active+clean; 92 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-03-11T18:40:40.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:40 smithi023 ceph-mon[36406]: pgmap v214: 129 pgs: 129 active+clean; 92 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-03-11T18:40:40.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:40 smithi002 ceph-mon[31328]: pgmap v214: 129 pgs: 129 active+clean; 92 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 -- 172.21.15.2:0/3170469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c81072b0 msgr2=0x7fc6c8107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 --2- 172.21.15.2:0/3170469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c81072b0 0x7fc6c8107690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009930 tx=0x7fc6b802bc70 comp rx=0 tx=0).stop 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 -- 172.21.15.2:0/3170469862 shutdown_connections 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 --2- 172.21.15.2:0/3170469862 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81003e0 0x7fc6c810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 --2- 172.21.15.2:0/3170469862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c80ffa20 0x7fc6c80ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 --2- 172.21.15.2:0/3170469862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c81072b0 0x7fc6c8107690 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 -- 172.21.15.2:0/3170469862 >> 172.21.15.2:0/3170469862 conn(0x7fc6c80753a0 msgr2=0x7fc6c80757b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 -- 172.21.15.2:0/3170469862 shutdown_connections 2024-03-11T18:40:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.296+0000 7fc6cee00640 1 -- 172.21.15.2:0/3170469862 wait complete. 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.297+0000 7fc6cee00640 1 Processor -- start 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 -- start start 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 0x7fc6c819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 0x7fc6c8199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c8199ec0 con 0x7fc6c81003e0 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c819a030 con 0x7fc6c80ffa20 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cee00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c819a1f0 con 0x7fc6c81072b0 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34580/0 (socket says 172.21.15.2:34580) 2024-03-11T18:40:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 0x7fc6c819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2415212435 learned_addr learned my addr 172.21.15.2:0/2415212435 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.298+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 0x7fc6c8199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 -- 172.21.15.2:0/2415212435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 msgr2=0x7fc6c819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 -- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 msgr2=0x7fc6c819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 0x7fc6c819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 -- 172.21.15.2:0/2415212435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b8009560 con 0x7fc6c81072b0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 0x7fc6c819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.299+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 0x7fc6c8199820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000e940 tx=0x7fc6c000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.300+0000 7fc6c4c00640 1 -- 172.21.15.2:0/2415212435 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0004d20 con 0x7fc6c81072b0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.300+0000 7fc6c4c00640 1 -- 172.21.15.2:0/2415212435 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6c0004e80 con 0x7fc6c81072b0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.300+0000 7fc6c4c00640 1 -- 172.21.15.2:0/2415212435 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00116d0 con 0x7fc6c81072b0 2024-03-11T18:40:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.300+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c819a4d0 con 0x7fc6c81072b0 2024-03-11T18:40:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.300+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c81a66e0 con 0x7fc6c81072b0 2024-03-11T18:40:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.304+0000 7fc6c4c00640 1 -- 172.21.15.2:0/2415212435 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fc6c00059c0 con 0x7fc6c81072b0 2024-03-11T18:40:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 Processor -- start 2024-03-11T18:40:41.307 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 -- start start 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de0e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de1a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79ddca620 con 0x55a79de0fb20 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd7c8c0 con 0x55a79de0edb0 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.305+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd79df0 con 0x55a79de18330 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34592/0 (socket says 172.21.15.2:34592) 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2515042631 learned_addr learned my addr 172.21.15.2:0/2515042631 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de0e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.309 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de1a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2515042631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 msgr2=0x55a79de1a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de1a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2515042631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 msgr2=0x55a79de0e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de0e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2515042631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a79dcf6b00 con 0x55a79de0edb0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de1a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c4c00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 0x7fc6ac0a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.306+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc6b802c1b0 tx=0x7fc6b8004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbf089d6ea13ab1e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.307+0000 7fc6c4c00640 1 -- 172.21.15.2:0/2415212435 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fc6c00cdc50 con 0x7fc6c81072b0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.307+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 0x7fc6ac0a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.307+0000 7fc6bd600640 1 -- 172.21.15.2:0/2515042631 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b802c840 con 0x55a79de0edb0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.307+0000 7fc6bd600640 1 -- 172.21.15.2:0/2515042631 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6b802de90 con 0x55a79de0edb0 2024-03-11T18:40:41.310 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.307+0000 7fc6bd600640 1 -- 172.21.15.2:0/2515042631 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80054c0 con 0x55a79de0edb0 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2515042631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 msgr2=0x55a79de0f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc6b802c1b0 tx=0x7fc6b8004af0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 0x7fc6ac0a40d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400b620 tx=0x7fc6b400bd50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2515042631 shutdown_connections 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de1a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de0f230 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2515042631 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de0e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2515042631 >> 172.21.15.2:0/2515042631 conn(0x55a79ddd64b0 msgr2=0x55a79de09aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2515042631 shutdown_connections 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.308+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2515042631 wait complete. 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 -- start start 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 Processor -- start 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 0x55a79de15350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de15dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd755d0 con 0x55a79de0edb0 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd1ab50 con 0x55a79de0fb20 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd1a7f0 con 0x55a79de18330 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6d1c18740 10 client.0 _ll_register_callbacks cb 0x55a79dd2e550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 0x55a79de15350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.309+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34604/0 (socket says 172.21.15.2:34604) 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7400640 1 -- 172.21.15.2:0/2666663177 learned_addr learned my addr 172.21.15.2:0/2666663177 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de15dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2666663177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 msgr2=0x55a79de15dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de15dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 msgr2=0x55a79de15890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2666663177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c810e8f0 con 0x55a79de0edb0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 0x55a79de15350 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8005ab0 tx=0x7fc6b8005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4198dfbfb0b0f5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.310+0000 7fc6b2000640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b804a650 con 0x55a79de0edb0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fc6b8004df0 con 0x55a79de0edb0 2024-03-11T18:40:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a79dd47cd0 con 0x55a79de0edb0 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6b804a7b0 con 0x55a79de0edb0 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6d1c18740 10 client.15894 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a79dd73c50 con 0x55a79de0edb0 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a79dd47cd0 con 0x55a79de0edb0 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.311+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc6b803c050 con 0x55a79de0edb0 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 10 client.15894 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 10 client.15894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 20 client.15894 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de27e40 0x55a79de2a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 4 client.15894 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a79dd6bce0 con 0x55a79de27e40 2024-03-11T18:40:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de27e40 0x55a79de2a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de27e40 0x55a79de2a250 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2489d3e09743c92c server_cookie=ed1c248c70da3b77 in_seq=0 out_seq=0 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.312+0000 7fc6b2000640 10 client.15894 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.313+0000 7fc6b2000640 1 -- 172.21.15.2:0/2666663177 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a79dd6bce0 con 0x55a79de27e40 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.313+0000 7fc6b2000640 10 client.15894 handle_command_reply: tid=0 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.313+0000 7fc6d1c18740 1 client.15894 shutdown 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 msgr2=0x55a79de15350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 0x55a79de15350 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8005ab0 tx=0x7fc6b8005470 comp rx=0 tx=0).stop 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 shutdown_connections 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de27e40 0x55a79de2a250 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de15dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0fb20 0x55a79de15890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2666663177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0edb0 0x55a79de15350 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 >> 172.21.15.2:0/2666663177 conn(0x55a79ddd64b0 msgr2=0x55a79de0b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6b2000640 0 client.15894 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 shutdown_connections 2024-03-11T18:40:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2666663177 wait complete. 2024-03-11T18:40:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.314+0000 7fc6d1c18740 20 client.15894 trim_cache size 0 max 16384 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 Processor -- start 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 -- start start 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 0x55a79de17a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de15bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dabde60 con 0x55a79de0fb20 2024-03-11T18:40:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79dd47cd0 con 0x55a79de18330 2024-03-11T18:40:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.332+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79de10f90 con 0x55a79de0edb0 2024-03-11T18:40:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 0x55a79de17a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34614/0 (socket says 172.21.15.2:34614) 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 -- 172.21.15.2:0/3288882540 learned_addr learned my addr 172.21.15.2:0/3288882540 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6c7400640 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de15bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 -- 172.21.15.2:0/3288882540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 msgr2=0x55a79de17a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 0x55a79de17a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 -- 172.21.15.2:0/3288882540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 msgr2=0x55a79de15bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de15bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 -- 172.21.15.2:0/3288882540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a79dd10530 con 0x55a79de18330 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 0x55a79de17a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6cce00640 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0096910 tx=0x7fc6c000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef492dab9d9fcb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.333+0000 7fc6bcc00640 1 -- 172.21.15.2:0/3288882540 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00ba070 con 0x55a79de18330 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6bcc00640 1 -- 172.21.15.2:0/3288882540 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6c009e550 con 0x55a79de18330 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6bcc00640 1 -- 172.21.15.2:0/3288882540 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00cb340 con 0x55a79de18330 2024-03-11T18:40:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 -- 172.21.15.2:0/3288882540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 msgr2=0x55a79de1f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0096910 tx=0x7fc6c000ef90 comp rx=0 tx=0).stop 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 -- 172.21.15.2:0/3288882540 shutdown_connections 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de18330 0x55a79de1f120 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de15bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/3288882540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de0edb0 0x55a79de17a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 -- 172.21.15.2:0/3288882540 >> 172.21.15.2:0/3288882540 conn(0x55a79ddd64b0 msgr2=0x55a79de0e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 -- 172.21.15.2:0/3288882540 shutdown_connections 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.334+0000 7fc6d1c18740 1 -- 172.21.15.2:0/3288882540 wait complete. 2024-03-11T18:40:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.335+0000 7fc6d1c18740 1 -- start start 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.335+0000 7fc6d1c18740 1 Processor -- start 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.335+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.335+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de11d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6d1c18740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de12280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79de12960 con 0x55a79de0fb20 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79de12ad0 con 0x55a79de0edb0 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6d1c18740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a79de7e310 con 0x55a79de18330 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6d1c18740 10 client.0 _ll_register_callbacks cb 0x7fc6b400bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de11d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:34626/0 (socket says 172.21.15.2:34626) 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2166987101 learned_addr learned my addr 172.21.15.2:0/2166987101 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6cce00640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de12280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2166987101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 msgr2=0x55a79de12280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de12280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2166987101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 msgr2=0x55a79de11d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de11d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 -- 172.21.15.2:0/2166987101 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a79dd73c50 con 0x55a79de0edb0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6c7e00640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8005ec0 tx=0x7fc6b8005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e8d901325ce339 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.336+0000 7fc6b2000640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b803fea0 con 0x55a79de0edb0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a79dd9ad30 con 0x55a79de0edb0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fc6b8000e40 con 0x55a79de0edb0 2024-03-11T18:40:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6b8005d50 con 0x55a79de0edb0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6d1c18740 10 client.26045 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fc6c81a66e0 con 0x55a79de0edb0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a79dd9ad30 con 0x55a79de0edb0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc6b803c040 con 0x55a79de0edb0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.337+0000 7fc6d1c18740 10 client.26045 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6d1c18740 10 client.26045 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6d1c18740 20 client.26045 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de7f670 0x55a79de81a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6d1c18740 4 client.26045 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a79da75600 con 0x55a79de7f670 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de7f670 0x55a79de81a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6c7400640 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de7f670 0x55a79de81a90 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9364aa6a9e502504 server_cookie=fcf4719534cc2b52 in_seq=0 out_seq=0 2024-03-11T18:40:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.338+0000 7fc6b2000640 10 client.26045 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:41.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 533319 8.06 MB/sec execute 221 sec latency 41.934 ms 2024-03-11T18:40:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.548+0000 7fc6b2000640 1 -- 172.21.15.2:0/2166987101 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a79da75600 con 0x55a79de7f670 2024-03-11T18:40:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.548+0000 7fc6b2000640 10 client.26045 handle_command_reply: tid=0 2024-03-11T18:40:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.548+0000 7fc6d1c18740 1 client.26045 shutdown 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 msgr2=0x55a79de1f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8005ec0 tx=0x7fc6b8005020 comp rx=0 tx=0).stop 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 shutdown_connections 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55a79de7f670 0x55a79de81a90 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55a79de18330 0x55a79de12280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a79de0fb20 0x55a79de11d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 --2- 172.21.15.2:0/2166987101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a79de0edb0 0x55a79de1f520 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 >> 172.21.15.2:0/2166987101 conn(0x55a79ddd64b0 msgr2=0x55a79de13be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6b2000640 0 client.26045 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 shutdown_connections 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 1 -- 172.21.15.2:0/2166987101 wait complete. 2024-03-11T18:40:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.549+0000 7fc6d1c18740 20 client.26045 trim_cache size 0 max 16384 2024-03-11T18:40:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:40:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:40:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:40:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:40:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 msgr2=0x7fc6ac0a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 0x7fc6ac0a40d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400b620 tx=0x7fc6b400bd50 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 msgr2=0x7fc6c8199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 0x7fc6c8199820 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000e940 tx=0x7fc6c000ee00 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 shutdown_connections 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6c81072b0 0x7fc6c8199820 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc6ac0a1c10 0x7fc6ac0a40d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc6c81003e0 0x7fc6c819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 --2- 172.21.15.2:0/2415212435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6c80ffa20 0x7fc6c819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 >> 172.21.15.2:0/2415212435 conn(0x7fc6c80753a0 msgr2=0x7fc6c810c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.550+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 shutdown_connections 2024-03-11T18:40:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:41.551+0000 7fc6cee00640 1 -- 172.21.15.2:0/2415212435 wait complete. 2024-03-11T18:40:42.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 534322 8.04 MB/sec execute 222 sec latency 77.471 ms 2024-03-11T18:40:42.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:42 smithi165 ceph-mon[36634]: pgmap v215: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-03-11T18:40:42.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:42 smithi023 ceph-mon[36406]: pgmap v215: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-03-11T18:40:42.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:42 smithi002 ceph-mon[31328]: pgmap v215: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-03-11T18:40:43.420 INFO:tasks.workunit.client.0.smithi002.stdout: 1 535868 8.04 MB/sec execute 223 sec latency 22.383 ms 2024-03-11T18:40:44.421 INFO:tasks.workunit.client.0.smithi002.stdout: 1 536979 8.03 MB/sec execute 224 sec latency 18.933 ms 2024-03-11T18:40:44.655 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:44 smithi165 ceph-mon[36634]: pgmap v216: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 138 op/s 2024-03-11T18:40:44.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:44 smithi023 ceph-mon[36406]: pgmap v216: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 138 op/s 2024-03-11T18:40:44.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:44 smithi002 ceph-mon[31328]: pgmap v216: 129 pgs: 129 active+clean; 93 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 1.6 MiB/s wr, 138 op/s 2024-03-11T18:40:45.421 INFO:tasks.workunit.client.0.smithi002.stdout: 1 537899 8.02 MB/sec execute 225 sec latency 59.804 ms 2024-03-11T18:40:45.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:45 smithi165 ceph-mon[36634]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2024-03-11T18:40:45.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:45 smithi023 ceph-mon[36406]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2024-03-11T18:40:45.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:45 smithi002 ceph-mon[31328]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2024-03-11T18:40:46.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 538302 7.99 MB/sec execute 226 sec latency 26.116 ms 2024-03-11T18:40:47.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 539805 7.99 MB/sec execute 227 sec latency 21.562 ms 2024-03-11T18:40:47.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:47 smithi165 ceph-mon[36634]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-03-11T18:40:47.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:47 smithi023 ceph-mon[36406]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-03-11T18:40:48.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:47 smithi002 ceph-mon[31328]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.8 MiB/s wr, 232 op/s 2024-03-11T18:40:48.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 540789 7.98 MB/sec execute 228 sec latency 24.068 ms 2024-03-11T18:40:49.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 542121 7.97 MB/sec execute 229 sec latency 24.474 ms 2024-03-11T18:40:49.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:49 smithi165 ceph-mon[36634]: pgmap v219: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-11T18:40:49.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:49 smithi023 ceph-mon[36406]: pgmap v219: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-11T18:40:50.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:49 smithi002 ceph-mon[31328]: pgmap v219: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-11T18:40:50.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 543606 7.98 MB/sec execute 230 sec latency 90.743 ms 2024-03-11T18:40:51.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 544464 7.96 MB/sec execute 231 sec latency 23.267 ms 2024-03-11T18:40:51.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:51 smithi165 ceph-mon[36634]: pgmap v220: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-03-11T18:40:51.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:51 smithi023 ceph-mon[36406]: pgmap v220: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-03-11T18:40:52.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:51 smithi002 ceph-mon[31328]: pgmap v220: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-03-11T18:40:52.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 545950 7.95 MB/sec execute 232 sec latency 40.624 ms 2024-03-11T18:40:53.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 547462 7.97 MB/sec execute 233 sec latency 12.853 ms 2024-03-11T18:40:53.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:53 smithi023 ceph-mon[36406]: pgmap v221: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:40:53.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:53 smithi165 ceph-mon[36634]: pgmap v221: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:40:54.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:53 smithi002 ceph-mon[31328]: pgmap v221: 129 pgs: 129 active+clean; 87 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:40:54.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 548589 7.96 MB/sec execute 234 sec latency 27.112 ms 2024-03-11T18:40:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 -- 172.21.15.2:0/1492656502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 msgr2=0x7f9920073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 --2- 172.21.15.2:0/1492656502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f9920073900 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9910009900 tx=0x7f991002bcd0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.698 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 -- 172.21.15.2:0/1492656502 shutdown_connections 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 --2- 172.21.15.2:0/1492656502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920112140 0x7f9920114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 --2- 172.21.15.2:0/1492656502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920073e40 0x7f9920111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 --2- 172.21.15.2:0/1492656502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f9920073900 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 -- 172.21.15.2:0/1492656502 >> 172.21.15.2:0/1492656502 conn(0x7f99200fcc00 msgr2=0x7f99200ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 -- 172.21.15.2:0/1492656502 shutdown_connections 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.695+0000 7f9926c00640 1 -- 172.21.15.2:0/1492656502 wait complete. 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.696+0000 7f9926c00640 1 Processor -- start 2024-03-11T18:40:54.699 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 -- start start 2024-03-11T18:40:54.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920073e40 0x7f992019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920112140 0x7f99201a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201a2ca0 con 0x7f9920073e40 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201a2e10 con 0x7f9920073520 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9926c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201a2fd0 con 0x7f9920112140 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f991fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f991fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55720/0 (socket says 172.21.15.2:55720) 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f991fe00640 1 -- 172.21.15.2:0/1688225869 learned_addr learned my addr 172.21.15.2:0/1688225869 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f991f400640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920073e40 0x7f992019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.697+0000 7f9924c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920112140 0x7f99201a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 -- 172.21.15.2:0/1688225869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920112140 msgr2=0x7f99201a26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920112140 0x7f99201a26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 -- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920073e40 msgr2=0x7f992019e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920073e40 0x7f992019e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 -- 172.21.15.2:0/1688225869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9910009560 con 0x7f9920073520 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991fe00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9910005940 tx=0x7f991002c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991cc00640 1 -- 172.21.15.2:0/1688225869 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910004350 con 0x7f9920073520 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991cc00640 1 -- 172.21.15.2:0/1688225869 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99100044b0 con 0x7f9920073520 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99201a3250 con 0x7f9920073520 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.698+0000 7f991cc00640 1 -- 172.21.15.2:0/1688225869 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910003d30 con 0x7f9920073520 2024-03-11T18:40:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.699+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99201aab30 con 0x7f9920073520 2024-03-11T18:40:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.703+0000 7f99299e1740 1 Processor -- start 2024-03-11T18:40:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.703+0000 7f99299e1740 1 -- start start 2024-03-11T18:40:54.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.703+0000 7f991cc00640 1 -- 172.21.15.2:0/1688225869 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f9910004df0 con 0x7f9920073520 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.703+0000 7f99299e1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f99299e1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca55230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f99299e1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 0x55b44ca60720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f99299e1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44ca10620 con 0x55b44ca5e330 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f99299e1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c9c28c0 con 0x55b44ca54db0 2024-03-11T18:40:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f99299e1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c9bfdf0 con 0x55b44ca55b20 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f991fe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca55230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:35600/0 (socket says 172.21.15.2:35600) 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 -- 172.21.15.2:0/1922605444 learned_addr learned my addr 172.21.15.2:0/1922605444 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f991f400640 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 0x55b44ca60720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 -- 172.21.15.2:0/1922605444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 msgr2=0x55b44ca55230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca55230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 -- 172.21.15.2:0/1922605444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 msgr2=0x55b44ca60720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 0x55b44ca60720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f9924c00640 1 -- 172.21.15.2:0/1922605444 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b44c93cb00 con 0x55b44ca55b20 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.704+0000 7f991f400640 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 0x55b44ca60720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f9924c00640 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9918009f30 tx=0x7f99180118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a69913ff7d7e3e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f991cc00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 0x7f99040a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f9915200640 1 -- 172.21.15.2:0/1922605444 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918004d20 con 0x55b44ca55b20 2024-03-11T18:40:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f9915200640 1 -- 172.21.15.2:0/1922605444 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9918004e80 con 0x55b44ca55b20 2024-03-11T18:40:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f9915200640 1 -- 172.21.15.2:0/1922605444 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918005280 con 0x55b44ca55b20 2024-03-11T18:40:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f991cc00640 1 -- 172.21.15.2:0/1688225869 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f99100f0200 con 0x7f9920073520 2024-03-11T18:40:54.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.705+0000 7f991fe00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 0x7f99040a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- 172.21.15.2:0/1922605444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 msgr2=0x55b44ca54870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9918009f30 tx=0x7f99180118d0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- 172.21.15.2:0/1922605444 shutdown_connections 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca5e330 0x55b44ca60720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca55230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.713 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 --2- 172.21.15.2:0/1922605444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca55b20 0x55b44ca54870 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- 172.21.15.2:0/1922605444 >> 172.21.15.2:0/1922605444 conn(0x55b44ca1c4b0 msgr2=0x55b44ca4faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f991fe00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 0x7f99040a4050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55b44ca56280 tx=0x7f99100f3c70 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- 172.21.15.2:0/1922605444 shutdown_connections 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- 172.21.15.2:0/1922605444 wait complete. 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.706+0000 7f99299e1740 1 -- start start 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 Processor -- start 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 0x55b44ca5b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 0x55b44ca5be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c9bb5d0 con 0x55b44ca55b20 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c960b50 con 0x55b44ca5e330 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c9607f0 con 0x55b44ca54db0 2024-03-11T18:40:54.714 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.707+0000 7f99299e1740 10 client.0 _ll_register_callbacks cb 0x55b44c974550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f9924c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 0x55b44ca5b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54118/0 (socket says 172.21.15.2:54118) 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f9924c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 0x55b44ca5b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:35602/0 (socket says 172.21.15.2:35602) 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 -- 172.21.15.2:0/3507839550 learned_addr learned my addr 172.21.15.2:0/3507839550 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f9924c00640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 0x55b44ca5be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 -- 172.21.15.2:0/3507839550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 msgr2=0x55b44ca5b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 0x55b44ca5b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 -- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 msgr2=0x55b44ca5be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 0x55b44ca5be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f991f400640 1 -- 172.21.15.2:0/3507839550 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9918009be0 con 0x55b44ca55b20 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.708+0000 7f9924c00640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 0x55b44ca5be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:40:54.715 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f991f400640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f990c012700 tx=0x7f990c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898ea09cbf6cdb61 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f9909600640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c017070 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b44c98dcd0 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f990c004050 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f990c004210 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f99299e1740 10 client.15912 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.709+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b44c9b9c50 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b44c98dcd0 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f990c003a00 con 0x55b44ca55b20 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 10 client.15912 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 20 client.15912 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44ca6e480 0x55b44ca708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.716 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 4 client.15912 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:40:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b44c9b1ce0 con 0x55b44ca6e480 2024-03-11T18:40:54.717 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.710+0000 7f9924c00640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44ca6e480 0x55b44ca708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.711+0000 7f9924c00640 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44ca6e480 0x55b44ca708a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81320606f18d6171 server_cookie=3c77c0a6c233e699 in_seq=0 out_seq=0 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.711+0000 7f9909600640 10 client.15912 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f9909600640 1 -- 172.21.15.2:0/3507839550 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b44c9b1ce0 con 0x55b44ca6e480 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f9909600640 10 client.15912 handle_command_reply: tid=0 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 client.15912 shutdown 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 msgr2=0x55b44ca5b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f990c012700 tx=0x7f990c012bc0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.718 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 shutdown_connections 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44ca6e480 0x55b44ca708a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca5e330 0x55b44ca5be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca5b930 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 --2- 172.21.15.2:0/3507839550 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca54db0 0x55b44ca5b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f9909600640 0 client.15912 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.712+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 >> 172.21.15.2:0/3507839550 conn(0x55b44ca1c4b0 msgr2=0x55b44ca517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.713+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 shutdown_connections 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.713+0000 7f99299e1740 1 -- 172.21.15.2:0/3507839550 wait complete. 2024-03-11T18:40:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.713+0000 7f99299e1740 20 client.15912 trim_cache size 0 max 16384 2024-03-11T18:40:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.727+0000 7f99299e1740 1 Processor -- start 2024-03-11T18:40:54.730 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.727+0000 7f99299e1740 1 -- start start 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca5db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca65150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44ca65690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44c703e60 con 0x55b44ca55b20 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44ca56110 con 0x55b44ca54db0 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f99299e1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44ca5b230 con 0x55b44ca5e330 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f9924c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca5db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f9924c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca5db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55760/0 (socket says 172.21.15.2:55760) 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca65150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f9924c00640 1 -- 172.21.15.2:0/2928757705 learned_addr learned my addr 172.21.15.2:0/2928757705 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f9924c00640 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44ca65690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 -- 172.21.15.2:0/2928757705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 msgr2=0x55b44ca65690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44ca65690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 -- 172.21.15.2:0/2928757705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 msgr2=0x55b44ca5db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca5db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.728+0000 7f991f400640 1 -- 172.21.15.2:0/2928757705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b44c956530 con 0x55b44ca55b20 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.729+0000 7f991f400640 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca65150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f990c003e10 tx=0x7f990c0096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e144dedf5c0a77c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.729+0000 7f990be00640 1 -- 172.21.15.2:0/2928757705 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c017070 con 0x55b44ca55b20 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.729+0000 7f990be00640 1 -- 172.21.15.2:0/2928757705 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f990c00f790 con 0x55b44ca55b20 2024-03-11T18:40:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.729+0000 7f990be00640 1 -- 172.21.15.2:0/2928757705 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c003a10 con 0x55b44ca55b20 2024-03-11T18:40:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- 172.21.15.2:0/2928757705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 msgr2=0x55b44ca65150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca65150 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f990c003e10 tx=0x7f990c0096b0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- 172.21.15.2:0/2928757705 shutdown_connections 2024-03-11T18:40:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44ca65690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca55b20 0x55b44ca65150 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 --2- 172.21.15.2:0/2928757705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca54db0 0x55b44ca5db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- 172.21.15.2:0/2928757705 >> 172.21.15.2:0/2928757705 conn(0x55b44ca1c4b0 msgr2=0x55b44ca59cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- 172.21.15.2:0/2928757705 shutdown_connections 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- 172.21.15.2:0/2928757705 wait complete. 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.730+0000 7f99299e1740 1 -- start start 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 Processor -- start 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca54db0 0x55b44ca586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44cac4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44ca59320 con 0x55b44ca54db0 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44cac4ec0 con 0x55b44ca55b20 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b44cac5030 con 0x55b44ca5e330 2024-03-11T18:40:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f99299e1740 10 client.0 _ll_register_callbacks cb 0x7f990c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f9924c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca54db0 0x55b44ca586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f991f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f991f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55776/0 (socket says 172.21.15.2:55776) 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f991f400640 1 -- 172.21.15.2:0/1635647803 learned_addr learned my addr 172.21.15.2:0/1635647803 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.731+0000 7f9924c00640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44cac4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 -- 172.21.15.2:0/1635647803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 msgr2=0x55b44cac4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44cac4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 -- 172.21.15.2:0/1635647803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca54db0 msgr2=0x55b44ca586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca54db0 0x55b44ca586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 -- 172.21.15.2:0/1635647803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c021040 con 0x55b44ca55b20 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f991f400640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f990c00a010 tx=0x7f990c00fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b941ee604a2677d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f9909600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c017040 con 0x55b44ca55b20 2024-03-11T18:40:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b44c9e0d30 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.732+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f990c000e40 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f990c00fe40 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 10 client.26063 fetch_fsmap learned FSMap version 29 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b44c97c2c0 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b44c9e0d30 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f990c030070 con 0x55b44ca55b20 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 10 client.26063 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 10 client.26063 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 20 client.26063 populate_metadata read hostname 'smithi002' 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44cac6470 0x55b44cac8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 4 client.26063 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b44c6bb600 con 0x55b44cac6470 2024-03-11T18:40:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.733+0000 7f9924c00640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44cac6470 0x55b44cac8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:40:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.734+0000 7f9924c00640 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44cac6470 0x55b44cac8860 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5848b1f938177fc7 server_cookie=9cbbc3be63737ce1 in_seq=0 out_seq=0 2024-03-11T18:40:54.740 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.734+0000 7f9909600640 10 client.26063 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.946+0000 7f9909600640 1 -- 172.21.15.2:0/1635647803 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b44c6bb600 con 0x55b44cac6470 2024-03-11T18:40:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.946+0000 7f9909600640 10 client.26063 handle_command_reply: tid=0 2024-03-11T18:40:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.946+0000 7f99299e1740 1 client.26063 shutdown 2024-03-11T18:40:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 msgr2=0x55b44ca58c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f990c00a010 tx=0x7f990c00fc50 comp rx=0 tx=0).stop 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 shutdown_connections 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b44cac6470 0x55b44cac8860 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b44ca5e330 0x55b44cac4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b44ca55b20 0x55b44ca58c20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 --2- 172.21.15.2:0/1635647803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b44ca54db0 0x55b44ca586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 >> 172.21.15.2:0/1635647803 conn(0x55b44ca1c4b0 msgr2=0x55b44ca4e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f9909600640 0 client.26063 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 shutdown_connections 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 1 -- 172.21.15.2:0/1635647803 wait complete. 2024-03-11T18:40:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.947+0000 7f99299e1740 20 client.26063 trim_cache size 0 max 16384 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 msgr2=0x7f99040a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 0x7f99040a4050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x55b44ca56280 tx=0x7f99100f3c70 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 msgr2=0x7f992019dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9910005940 tx=0x7f991002c710 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 shutdown_connections 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9920112140 0x7f99201a26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f99040a1b90 0x7f99040a4050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9920073e40 0x7f992019e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 --2- 172.21.15.2:0/1688225869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9920073520 0x7f992019dde0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 >> 172.21.15.2:0/1688225869 conn(0x7f99200fcc00 msgr2=0x7f99200fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 shutdown_connections 2024-03-11T18:40:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:40:54.948+0000 7f9926c00640 1 -- 172.21.15.2:0/1688225869 wait complete. 2024-03-11T18:40:55.422 INFO:tasks.workunit.client.0.smithi002.stdout: 1 550001 7.96 MB/sec execute 235 sec latency 85.648 ms 2024-03-11T18:40:55.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:55 smithi165 ceph-mon[36634]: pgmap v222: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-03-11T18:40:55.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:55 smithi023 ceph-mon[36406]: pgmap v222: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-03-11T18:40:56.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:55 smithi002 ceph-mon[31328]: pgmap v222: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-03-11T18:40:56.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 551324 7.95 MB/sec execute 236 sec latency 25.127 ms 2024-03-11T18:40:57.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 552474 7.94 MB/sec execute 237 sec latency 41.503 ms 2024-03-11T18:40:57.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:57 smithi165 ceph-mon[36634]: pgmap v223: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-03-11T18:40:57.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:57 smithi023 ceph-mon[36406]: pgmap v223: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-03-11T18:40:58.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:57 smithi002 ceph-mon[31328]: pgmap v223: 129 pgs: 129 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.9 MiB/s wr, 152 op/s 2024-03-11T18:40:58.425 INFO:tasks.workunit.client.0.smithi002.stdout: 1 554421 7.96 MB/sec execute 238 sec latency 25.967 ms 2024-03-11T18:40:59.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 555570 7.95 MB/sec execute 239 sec latency 25.242 ms 2024-03-11T18:40:59.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:40:59 smithi165 ceph-mon[36634]: pgmap v224: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-03-11T18:40:59.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:40:59 smithi023 ceph-mon[36406]: pgmap v224: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-03-11T18:41:00.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:40:59 smithi002 ceph-mon[31328]: pgmap v224: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-03-11T18:41:00.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 557141 7.95 MB/sec execute 240 sec latency 28.609 ms 2024-03-11T18:41:01.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 557416 7.92 MB/sec execute 241 sec latency 117.431 ms 2024-03-11T18:41:01.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:01 smithi165 ceph-mon[36634]: pgmap v225: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-03-11T18:41:01.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:41:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:01 smithi023 ceph-mon[36406]: pgmap v225: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-03-11T18:41:01.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:41:02.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:01 smithi002 ceph-mon[31328]: pgmap v225: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 1.8 MiB/s wr, 240 op/s 2024-03-11T18:41:02.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:41:02.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 558180 7.91 MB/sec execute 242 sec latency 111.922 ms 2024-03-11T18:41:03.423 INFO:tasks.workunit.client.0.smithi002.stdout: 1 558592 7.88 MB/sec execute 243 sec latency 62.743 ms 2024-03-11T18:41:04.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:03 smithi002 ceph-mon[31328]: pgmap v226: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-03-11T18:41:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:03 smithi002 ceph-mon[31328]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T18:41:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:03 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:41:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:03 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:41:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:03 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:03 smithi165 ceph-mon[36634]: pgmap v226: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:03 smithi165 ceph-mon[36634]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:03 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:03 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:03 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:03 smithi023 ceph-mon[36406]: pgmap v226: 129 pgs: 129 active+clean; 87 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.6 MiB/s wr, 226 op/s 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:03 smithi023 ceph-mon[36406]: osdmap e100: 12 total, 12 up, 12 in 2024-03-11T18:41:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:03 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:41:04.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:03 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:41:04.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:03 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:41:04.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 559343 7.87 MB/sec execute 244 sec latency 113.929 ms 2024-03-11T18:41:05.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 560094 7.84 MB/sec execute 245 sec latency 117.148 ms 2024-03-11T18:41:06.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:05 smithi002 ceph-mon[31328]: pgmap v228: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:05 smithi165 ceph-mon[36634]: pgmap v228: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:06.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:05 smithi023 ceph-mon[36406]: pgmap v228: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:06.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 561231 7.85 MB/sec execute 246 sec latency 40.878 ms 2024-03-11T18:41:07.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 561993 7.83 MB/sec execute 247 sec latency 25.546 ms 2024-03-11T18:41:08.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:07 smithi002 ceph-mon[31328]: pgmap v229: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 -- 172.21.15.2:0/1031451204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 msgr2=0x7fa48c10af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 --2- 172.21.15.2:0/1031451204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c10af30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa474009900 tx=0x7fa47402bcc0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 -- 172.21.15.2:0/1031451204 shutdown_connections 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 --2- 172.21.15.2:0/1031451204 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c10b550 0x7fa48c10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 --2- 172.21.15.2:0/1031451204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c10af30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 --2- 172.21.15.2:0/1031451204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c102130 0x7fa48c102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 -- 172.21.15.2:0/1031451204 >> 172.21.15.2:0/1031451204 conn(0x7fa48c0fb970 msgr2=0x7fa48c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 -- 172.21.15.2:0/1031451204 shutdown_connections 2024-03-11T18:41:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.103+0000 7fa490a00640 1 -- 172.21.15.2:0/1031451204 wait complete. 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.104+0000 7fa490a00640 1 Processor -- start 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.104+0000 7fa490a00640 1 -- start start 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 0x7fa48c0ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c1002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c10b550 0x7fa48c100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a5a10 con 0x7fa48c102130 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a5b80 con 0x7fa48c102a50 2024-03-11T18:41:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa490a00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a5d40 con 0x7fa48c10b550 2024-03-11T18:41:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa48a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 0x7fa48c0ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa489e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c1002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa48a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 0x7fa48c0ffd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56666/0 (socket says 172.21.15.2:56666) 2024-03-11T18:41:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa48a800640 1 -- 172.21.15.2:0/1178642824 learned_addr learned my addr 172.21.15.2:0/1178642824 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa48b200640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c10b550 0x7fa48c100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa489e00640 1 -- 172.21.15.2:0/1178642824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c10b550 msgr2=0x7fa48c100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa489e00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c10b550 0x7fa48c100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa489e00640 1 -- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 msgr2=0x7fa48c0ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.105+0000 7fa489e00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 0x7fa48c0ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.106+0000 7fa489e00640 1 -- 172.21.15.2:0/1178642824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474009560 con 0x7fa48c102a50 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.106+0000 7fa489e00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c1002c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa47402c200 tx=0x7fa47402c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.106+0000 7fa483400640 1 -- 172.21.15.2:0/1178642824 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47402db10 con 0x7fa48c102a50 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.106+0000 7fa483400640 1 -- 172.21.15.2:0/1178642824 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa47402dc70 con 0x7fa48c102a50 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.107+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c1a5fc0 con 0x7fa48c102a50 2024-03-11T18:41:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.107+0000 7fa483400640 1 -- 172.21.15.2:0/1178642824 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474003af0 con 0x7fa48c102a50 2024-03-11T18:41:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.107+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c1a6490 con 0x7fa48c102a50 2024-03-11T18:41:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.110+0000 7fa483400640 1 -- 172.21.15.2:0/1178642824 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fa4740053d0 con 0x7fa48c102a50 2024-03-11T18:41:08.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.111+0000 7fa494194740 1 Processor -- start 2024-03-11T18:41:08.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.111+0000 7fa494194740 1 -- start start 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 0x55562bac4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bacd570 0x55562bacf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562ba7f620 con 0x55562bac3ff0 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562ba318c0 con 0x55562bac4d60 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa494194740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562ba2edf0 con 0x55562bacd570 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:35066/0 (socket says 172.21.15.2:35066) 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa489e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bacd570 0x55562bacf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 -- 172.21.15.2:0/2906434479 learned_addr learned my addr 172.21.15.2:0/2906434479 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48a800640 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 0x55562bac4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 -- 172.21.15.2:0/2906434479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bacd570 msgr2=0x55562bacf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bacd570 0x55562bacf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 -- 172.21.15.2:0/2906434479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 msgr2=0x55562bac4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 0x55562bac4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48b200640 1 -- 172.21.15.2:0/2906434479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55562b9abb00 con 0x55562bac4d60 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.112+0000 7fa48a800640 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 0x55562bac4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa48b200640 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa484009f70 tx=0x7fa484010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba3ee1290367dd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa483400640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 0x7fa4700a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa483400640 1 -- 172.21.15.2:0/1178642824 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fa4740eff90 con 0x7fa48c102a50 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa47b400640 1 -- 172.21.15.2:0/2906434479 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484004d20 con 0x55562bac4d60 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa47b400640 1 -- 172.21.15.2:0/2906434479 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa484004e80 con 0x55562bac4d60 2024-03-11T18:41:08.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa47b400640 1 -- 172.21.15.2:0/2906434479 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4840129b0 con 0x55562bac4d60 2024-03-11T18:41:08.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.113+0000 7fa48a800640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 0x7fa4700a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 -- 172.21.15.2:0/2906434479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 msgr2=0x55562bac3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa48a800640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 0x7fa4700a40f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55562bac54c0 tx=0x7fa47c005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa484009f70 tx=0x7fa484010800 comp rx=0 tx=0).stop 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 -- 172.21.15.2:0/2906434479 shutdown_connections 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bacd570 0x55562bacf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac3ff0 0x55562bac4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 --2- 172.21.15.2:0/2906434479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac3ab0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 -- 172.21.15.2:0/2906434479 >> 172.21.15.2:0/2906434479 conn(0x55562ba8b4b0 msgr2=0x55562babece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 -- 172.21.15.2:0/2906434479 shutdown_connections 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.114+0000 7fa494194740 1 -- 172.21.15.2:0/2906434479 wait complete. 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.115+0000 7fa494194740 1 -- start start 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.115+0000 7fa494194740 1 Processor -- start 2024-03-11T18:41:08.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac3ff0 0x55562baca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bacb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562ba2a5d0 con 0x55562bacd570 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562b9cfb50 con 0x55562bac3ff0 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562b9cf7f0 con 0x55562bac4d60 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa494194740 10 client.0 _ll_register_callbacks cb 0x55562b9e3550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa48a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac3ff0 0x55562baca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47874/0 (socket says 172.21.15.2:47874) 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 -- 172.21.15.2:0/1405122777 learned_addr learned my addr 172.21.15.2:0/1405122777 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa48b200640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bacb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 -- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac3ff0 msgr2=0x55562baca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac3ff0 0x55562baca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 -- 172.21.15.2:0/1405122777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 msgr2=0x55562bacb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bacb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.116+0000 7fa489e00640 1 -- 172.21.15.2:0/1405122777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa484009bd0 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa489e00640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa47402c810 tx=0x7fa4740b8ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a81bfb1eed443c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa46fe00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474003e50 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fa4740bcf20 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55562b9fccd0 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4740bc180 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa494194740 10 client.26065 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55562ba28c50 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.117+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55562b9fccd0 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa47403c070 con 0x55562bac4d60 2024-03-11T18:41:08.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 10 client.26065 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 10 client.26065 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 20 client.26065 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562badd460 0x55562badf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 4 client.26065 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55562ba20ce0 con 0x55562badd460 2024-03-11T18:41:08.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa48a800640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562badd460 0x55562badf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa48a800640 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562badd460 0x55562badf850 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c8734c00f2937b server_cookie=e8cb02552ce3ec7e in_seq=0 out_seq=0 2024-03-11T18:41:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.118+0000 7fa46fe00640 10 client.26065 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.119+0000 7fa46fe00640 1 -- 172.21.15.2:0/1405122777 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55562ba20ce0 con 0x55562badd460 2024-03-11T18:41:08.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.119+0000 7fa46fe00640 10 client.26065 handle_command_reply: tid=0 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 client.26065 shutdown 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 msgr2=0x55562bacab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa47402c810 tx=0x7fa4740b8ac0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 shutdown_connections 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562badd460 0x55562badf850 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bacb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac4d60 0x55562bacab50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 --2- 172.21.15.2:0/1405122777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac3ff0 0x55562baca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa46fe00640 0 client.26065 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 >> 172.21.15.2:0/1405122777 conn(0x55562ba8b4b0 msgr2=0x55562bac0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 shutdown_connections 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 1 -- 172.21.15.2:0/1405122777 wait complete. 2024-03-11T18:41:08.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.120+0000 7fa494194740 20 client.26065 trim_cache size 0 max 16384 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.136+0000 7fa494194740 1 Processor -- start 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.136+0000 7fa494194740 1 -- start start 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bac9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562b772e60 con 0x55562bacd570 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562bac5350 con 0x55562bac4d60 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa494194740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562baba5a0 con 0x55562bac3ff0 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa489e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47886/0 (socket says 172.21.15.2:47886) 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 -- 172.21.15.2:0/225556417 learned_addr learned my addr 172.21.15.2:0/225556417 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48b200640 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bac9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 -- 172.21.15.2:0/225556417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 msgr2=0x55562bac8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 -- 172.21.15.2:0/225556417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 msgr2=0x55562bac9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bac9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.140 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48a800640 1 -- 172.21.15.2:0/225556417 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55562b9c5530 con 0x55562bac3ff0 2024-03-11T18:41:08.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.137+0000 7fa48b200640 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bac9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.138+0000 7fa48a800640 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55562bae0160 tx=0x7fa47c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cbec90c1007f2aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.138+0000 7fa47aa00640 1 -- 172.21.15.2:0/225556417 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c023030 con 0x55562bac3ff0 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.138+0000 7fa47aa00640 1 -- 172.21.15.2:0/225556417 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa47c024040 con 0x55562bac3ff0 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.138+0000 7fa47aa00640 1 -- 172.21.15.2:0/225556417 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c017070 con 0x55562bac3ff0 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- 172.21.15.2:0/225556417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 msgr2=0x55562baccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55562bae0160 tx=0x7fa47c019000 comp rx=0 tx=0).stop 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- 172.21.15.2:0/225556417 shutdown_connections 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bacd570 0x55562bac9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bac4d60 0x55562bac8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 --2- 172.21.15.2:0/225556417 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562baccd40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- 172.21.15.2:0/225556417 >> 172.21.15.2:0/225556417 conn(0x55562ba8b4b0 msgr2=0x55562babe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- 172.21.15.2:0/225556417 shutdown_connections 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- 172.21.15.2:0/225556417 wait complete. 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.139+0000 7fa494194740 1 -- start start 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 Processor -- start 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac4d60 0x55562bad3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bacd570 0x55562bad3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562bad4cc0 con 0x55562bac4d60 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562bb33940 con 0x55562bacd570 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562bb33a80 con 0x55562bac3ff0 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa494194740 10 client.0 _ll_register_callbacks cb 0x7fa474009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa48a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa48a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47902/0 (socket says 172.21.15.2:47902) 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa48a800640 1 -- 172.21.15.2:0/3498748283 learned_addr learned my addr 172.21.15.2:0/3498748283 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa489e00640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac4d60 0x55562bad3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.140+0000 7fa48b200640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bacd570 0x55562bad3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 -- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bacd570 msgr2=0x55562bad3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bacd570 0x55562bad3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 -- 172.21.15.2:0/3498748283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac4d60 msgr2=0x55562bad3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac4d60 0x55562bad3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 -- 172.21.15.2:0/3498748283 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c01d530 con 0x55562bac3ff0 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa48a800640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009620 tx=0x7fa47c02a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ad61c295268692 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:08.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa46fe00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:08.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.141+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c023070 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fa47c005230 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55562ba4fd30 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa47c017070 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa494194740 10 client.26071 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55562b9eb2c0 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55562ba4fd30 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.142+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa47c0335b0 con 0x55562bac3ff0 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 10 client.26071 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 10 client.26071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 20 client.26071 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562bb351f0 0x55562bb37610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 4 client.26071 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:41:08.147 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55562b72a600 con 0x55562bb351f0 2024-03-11T18:41:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa489e00640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562bb351f0 0x55562bb37610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa489e00640 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562bb351f0 0x55562bb37610 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff04a011156467a server_cookie=88419b0ff55259ef in_seq=0 out_seq=0 2024-03-11T18:41:08.148 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.143+0000 7fa46fe00640 10 client.26071 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:07 smithi165 ceph-mon[36634]: pgmap v229: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:07 smithi023 ceph-mon[36406]: pgmap v229: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 90 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.5 MiB/s wr, 281 op/s 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.317+0000 7fa46fe00640 1 -- 172.21.15.2:0/3498748283 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55562b72a600 con 0x55562bb351f0 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.317+0000 7fa46fe00640 10 client.26071 handle_command_reply: tid=0 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.317+0000 7fa494194740 1 client.26071 shutdown 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 msgr2=0x55562bad2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009620 tx=0x7fa47c02a620 comp rx=0 tx=0).stop 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 shutdown_connections 2024-03-11T18:41:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55562bb351f0 0x55562bb37610 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55562bacd570 0x55562bad3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55562bac4d60 0x55562bad3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 --2- 172.21.15.2:0/3498748283 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55562bac3ff0 0x55562bad2dc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa46fe00640 0 client.26071 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 >> 172.21.15.2:0/3498748283 conn(0x55562ba8b4b0 msgr2=0x55562babe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 shutdown_connections 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 1 -- 172.21.15.2:0/3498748283 wait complete. 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.318+0000 7fa494194740 20 client.26071 trim_cache size 0 max 16384 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:41:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:41:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:41:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:41:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 msgr2=0x7fa4700a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 0x7fa4700a40f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55562bac54c0 tx=0x7fa47c005f90 comp rx=0 tx=0).stop 2024-03-11T18:41:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 msgr2=0x7fa48c1002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c1002c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa47402c200 tx=0x7fa47402c680 comp rx=0 tx=0).stop 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 shutdown_connections 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa48c10b550 0x7fa48c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c102a50 0x7fa48c1002c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa4700a1c30 0x7fa4700a40f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 --2- 172.21.15.2:0/1178642824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa48c102130 0x7fa48c0ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 >> 172.21.15.2:0/1178642824 conn(0x7fa48c0fb970 msgr2=0x7fa48c0fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 shutdown_connections 2024-03-11T18:41:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:08.319+0000 7fa490a00640 1 -- 172.21.15.2:0/1178642824 wait complete. 2024-03-11T18:41:08.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 562913 7.82 MB/sec execute 248 sec latency 100.006 ms 2024-03-11T18:41:09.424 INFO:tasks.workunit.client.0.smithi002.stdout: 1 564640 7.82 MB/sec execute 249 sec latency 23.329 ms 2024-03-11T18:41:10.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:09 smithi002 ceph-mon[31328]: pgmap v230: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 83 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 1.3 MiB/s wr, 221 op/s 2024-03-11T18:41:10.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:09 smithi165 ceph-mon[36634]: pgmap v230: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 83 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 1.3 MiB/s wr, 221 op/s 2024-03-11T18:41:10.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:09 smithi023 ceph-mon[36406]: pgmap v230: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 83 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 1.3 MiB/s wr, 221 op/s 2024-03-11T18:41:10.426 INFO:tasks.workunit.client.0.smithi002.stdout: 1 565480 7.81 MB/sec execute 250 sec latency 99.377 ms 2024-03-11T18:41:11.426 INFO:tasks.workunit.client.0.smithi002.stdout: 1 566620 7.80 MB/sec execute 251 sec latency 24.292 ms 2024-03-11T18:41:12.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:11 smithi002 ceph-mon[31328]: pgmap v231: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:12.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:11 smithi165 ceph-mon[36634]: pgmap v231: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:12.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:11 smithi023 ceph-mon[36406]: pgmap v231: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:12.427 INFO:tasks.workunit.client.0.smithi002.stdout: 1 568565 7.82 MB/sec execute 252 sec latency 21.008 ms 2024-03-11T18:41:13.427 INFO:tasks.workunit.client.0.smithi002.stdout: 1 569480 7.80 MB/sec execute 253 sec latency 25.498 ms 2024-03-11T18:41:14.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:13 smithi002 ceph-mon[31328]: pgmap v232: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:14.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:13 smithi002 ceph-mon[31328]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T18:41:14.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:13 smithi165 ceph-mon[36634]: pgmap v232: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:14.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:13 smithi165 ceph-mon[36634]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T18:41:14.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:13 smithi023 ceph-mon[36406]: pgmap v232: 129 pgs: 129 active+clean; 86 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 247 op/s 2024-03-11T18:41:14.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:13 smithi023 ceph-mon[36406]: osdmap e101: 12 total, 12 up, 12 in 2024-03-11T18:41:14.427 INFO:tasks.workunit.client.0.smithi002.stdout: 1 570897 7.79 MB/sec execute 254 sec latency 32.744 ms 2024-03-11T18:41:15.427 INFO:tasks.workunit.client.0.smithi002.stdout: 1 572330 7.81 MB/sec execute 255 sec latency 30.343 ms 2024-03-11T18:41:16.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:15 smithi002 ceph-mon[31328]: pgmap v234: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:16.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:15 smithi165 ceph-mon[36634]: pgmap v234: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:16.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:15 smithi023 ceph-mon[36406]: pgmap v234: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:16.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 573546 7.80 MB/sec execute 256 sec latency 23.619 ms 2024-03-11T18:41:17.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 575521 7.82 MB/sec execute 257 sec latency 20.491 ms 2024-03-11T18:41:17.957 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:17 smithi165 ceph-mon[36634]: pgmap v235: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:18.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:17 smithi002 ceph-mon[31328]: pgmap v235: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:17 smithi023 ceph-mon[36406]: pgmap v235: 129 pgs: 129 active+clean; 90 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 243 op/s 2024-03-11T18:41:18.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 576526 7.80 MB/sec execute 258 sec latency 43.280 ms 2024-03-11T18:41:19.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 578195 7.80 MB/sec execute 259 sec latency 26.686 ms 2024-03-11T18:41:20.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:19 smithi002 ceph-mon[31328]: pgmap v236: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-03-11T18:41:20.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:19 smithi165 ceph-mon[36634]: pgmap v236: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-03-11T18:41:20.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:19 smithi023 ceph-mon[36406]: pgmap v236: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 216 op/s 2024-03-11T18:41:20.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 579521 7.80 MB/sec execute 260 sec latency 28.713 ms 2024-03-11T18:41:21.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 580669 7.79 MB/sec execute 261 sec latency 21.761 ms 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 -- 172.21.15.2:0/2945721938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 msgr2=0x7f906c068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 --2- 172.21.15.2:0/2945721938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c068fc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f905c009930 tx=0x7f905c02bc30 comp rx=0 tx=0).stop 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 -- 172.21.15.2:0/2945721938 shutdown_connections 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 --2- 172.21.15.2:0/2945721938 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 0x7f906c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 --2- 172.21.15.2:0/2945721938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c069590 0x7f906c105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 --2- 172.21.15.2:0/2945721938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c068fc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.470+0000 7f906b200640 1 -- 172.21.15.2:0/2945721938 >> 172.21.15.2:0/2945721938 conn(0x7f906c0759e0 msgr2=0x7f906c075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.471+0000 7f906b200640 1 -- 172.21.15.2:0/2945721938 shutdown_connections 2024-03-11T18:41:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.471+0000 7f906b200640 1 -- 172.21.15.2:0/2945721938 wait complete. 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.471+0000 7f906b200640 1 Processor -- start 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 -- start start 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 0x7f906c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 0x7f906c19c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c19c8d0 con 0x7f906c19bee0 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c19ca40 con 0x7f906c068be0 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906c19cbb0 con 0x7f906c106370 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f9069e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f9069e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39222/0 (socket says 172.21.15.2:39222) 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f9069e00640 1 -- 172.21.15.2:0/1605907922 learned_addr learned my addr 172.21.15.2:0/1605907922 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f906a800640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 0x7f906c19c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.472+0000 7f9061400640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 0x7f906c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 -- 172.21.15.2:0/1605907922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 msgr2=0x7f906c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 0x7f906c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 -- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 msgr2=0x7f906c19c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 0x7f906c19c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 -- 172.21.15.2:0/1605907922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c009560 con 0x7f906c068be0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f906a800640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 0x7f906c19c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9069e00640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f905c000c00 tx=0x7f905c02c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9063400640 1 -- 172.21.15.2:0/1605907922 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c0042d0 con 0x7f906c068be0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.473+0000 7f9063400640 1 -- 172.21.15.2:0/1605907922 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f905c004430 con 0x7f906c068be0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.474+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906c1a4190 con 0x7f906c068be0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.474+0000 7f9063400640 1 -- 172.21.15.2:0/1605907922 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c003c20 con 0x7f906c068be0 2024-03-11T18:41:21.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.474+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906c1a4640 con 0x7f906c068be0 2024-03-11T18:41:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 Processor -- start 2024-03-11T18:41:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9063400640 1 -- 172.21.15.2:0/1605907922 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f905c004df0 con 0x7f906c068be0 2024-03-11T18:41:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 -- start start 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f94f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f958570 0x56408f95a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f90a620 con 0x56408f958570 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f8bc8c0 con 0x56408f94fd60 2024-03-11T18:41:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.477+0000 7f9073636740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f8b9df0 con 0x56408f94eff0 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39224/0 (socket says 172.21.15.2:39224) 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 -- 172.21.15.2:0/4095545790 learned_addr learned my addr 172.21.15.2:0/4095545790 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f9069e00640 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f94f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 -- 172.21.15.2:0/4095545790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 msgr2=0x56408f94f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f94f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 -- 172.21.15.2:0/4095545790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f958570 msgr2=0x56408f95a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f958570 0x56408f95a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 -- 172.21.15.2:0/4095545790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56408f836b00 con 0x56408f94fd60 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.478+0000 7f906a800640 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9064009fd0 tx=0x7f90640108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a567249b978ede4d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f9063400640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 0x7f90500a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f905b000640 1 -- 172.21.15.2:0/4095545790 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064011760 con 0x56408f94fd60 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f905b000640 1 -- 172.21.15.2:0/4095545790 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9064015070 con 0x56408f94fd60 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f905b000640 1 -- 172.21.15.2:0/4095545790 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906401be00 con 0x56408f94fd60 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f9063400640 1 -- 172.21.15.2:0/1605907922 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f905c0f0190 con 0x7f906c068be0 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.479+0000 7f9069e00640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 0x7f90500a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 -- 172.21.15.2:0/4095545790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 msgr2=0x56408f94eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9064009fd0 tx=0x7f90640108e0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 -- 172.21.15.2:0/4095545790 shutdown_connections 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f958570 0x56408f95a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f94f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 --2- 172.21.15.2:0/4095545790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f94fd60 0x56408f94eab0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 -- 172.21.15.2:0/4095545790 >> 172.21.15.2:0/4095545790 conn(0x56408f9164b0 msgr2=0x56408f949ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 -- 172.21.15.2:0/4095545790 shutdown_connections 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9073636740 1 -- 172.21.15.2:0/4095545790 wait complete. 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.480+0000 7f9069e00640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 0x7f90500a40c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x56408f9504c0 tx=0x7f905c0f3c00 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.481+0000 7f9073636740 1 -- start start 2024-03-11T18:41:21.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.481+0000 7f9073636740 1 Processor -- start 2024-03-11T18:41:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.481+0000 7f9073636740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f94eff0 0x56408f9624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.481+0000 7f9073636740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9073636740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9073636740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f8b55d0 con 0x56408f94eff0 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9073636740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f85ab50 con 0x56408f958570 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9073636740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f85a7f0 con 0x56408f94fd60 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9073636740 10 client.0 _ll_register_callbacks cb 0x56408f86e550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:44672/0 (socket says 172.21.15.2:44672) 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 -- 172.21.15.2:0/1522704094 learned_addr learned my addr 172.21.15.2:0/1522704094 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f906a800640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f94eff0 0x56408f9624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 -- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 msgr2=0x56408f95c8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f906a800640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 -- 172.21.15.2:0/1522704094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f94eff0 msgr2=0x56408f9624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f94eff0 0x56408f9624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f9061400640 1 -- 172.21.15.2:0/1522704094 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9064009cb0 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.482+0000 7f906a800640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f9061400640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f906c102330 tx=0x7f9054011920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d37ab91a6e357a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054004520 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56408f887cd0 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f90540035f0 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90540037b0 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f9073636740 10 client.26083 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56408f8b3c50 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56408f887cd0 con 0x56408f94fd60 2024-03-11T18:41:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.483+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f905400fbc0 con 0x56408f94fd60 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 10 client.26083 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 10 client.26083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 20 client.26083 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f96ca40 0x56408f96ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 4 client.26083 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56408f8abce0 con 0x56408f96ca40 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f906a800640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f96ca40 0x56408f96ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f906a800640 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f96ca40 0x56408f96ee60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32352790b69d1811 server_cookie=65b2c79ee1d57fe0 in_seq=0 out_seq=0 2024-03-11T18:41:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.484+0000 7f904f400640 10 client.26083 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f904f400640 1 -- 172.21.15.2:0/1522704094 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56408f8abce0 con 0x56408f96ca40 2024-03-11T18:41:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f904f400640 10 client.26083 handle_command_reply: tid=0 2024-03-11T18:41:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 client.26083 shutdown 2024-03-11T18:41:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 msgr2=0x56408f9629e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f906c102330 tx=0x7f9054011920 comp rx=0 tx=0).stop 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 shutdown_connections 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f96ca40 0x56408f96ee60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94fd60 0x56408f9629e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 --2- 172.21.15.2:0/1522704094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f94eff0 0x56408f9624a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f904f400640 0 client.26083 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 >> 172.21.15.2:0/1522704094 conn(0x56408f9164b0 msgr2=0x56408f94ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 shutdown_connections 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.486+0000 7f9073636740 1 -- 172.21.15.2:0/1522704094 wait complete. 2024-03-11T18:41:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.487+0000 7f9073636740 20 client.26083 trim_cache size 0 max 16384 2024-03-11T18:41:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 Processor -- start 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 -- start start 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f95c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f962000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f5fde60 con 0x56408f949f20 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f96b2e0 con 0x56408f958570 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.502+0000 7f9073636740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f8abce0 con 0x56408f94eff0 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f906a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f962000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:39238/0 (socket says 172.21.15.2:39238) 2024-03-11T18:41:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 -- 172.21.15.2:0/2765422492 learned_addr learned my addr 172.21.15.2:0/2765422492 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f906a800640 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f95c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 -- 172.21.15.2:0/2765422492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 msgr2=0x56408f95c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f95c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 -- 172.21.15.2:0/2765422492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 msgr2=0x56408f962000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f962000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 -- 172.21.15.2:0/2765422492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56408f850530 con 0x56408f958570 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f906a800640 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f962000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.503+0000 7f9061400640 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9054010010 tx=0x7f9054011ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e204272df543559f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f905a600640 1 -- 172.21.15.2:0/2765422492 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054019570 con 0x56408f958570 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f905a600640 1 -- 172.21.15.2:0/2765422492 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9054019e80 con 0x56408f958570 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f905a600640 1 -- 172.21.15.2:0/2765422492 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905400f940 con 0x56408f958570 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 -- 172.21.15.2:0/2765422492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 msgr2=0x56408f95c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9054010010 tx=0x7f9054011ea0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 -- 172.21.15.2:0/2765422492 shutdown_connections 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f962000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f95c680 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 --2- 172.21.15.2:0/2765422492 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f95c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.504+0000 7f9073636740 1 -- 172.21.15.2:0/2765422492 >> 172.21.15.2:0/2765422492 conn(0x56408f9164b0 msgr2=0x56408f949810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.505+0000 7f9073636740 1 -- 172.21.15.2:0/2765422492 shutdown_connections 2024-03-11T18:41:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.505+0000 7f9073636740 1 -- 172.21.15.2:0/2765422492 wait complete. 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.505+0000 7f9073636740 1 -- start start 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.505+0000 7f9073636740 1 Processor -- start 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f9bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f9b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f9b7570 con 0x56408f949f20 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f9b76e0 con 0x56408f958570 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56408f9b78a0 con 0x56408f94eff0 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9073636740 10 client.0 _ll_register_callbacks cb 0x7f9054009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:44700/0 (socket says 172.21.15.2:44700) 2024-03-11T18:41:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 -- 172.21.15.2:0/1683228698 learned_addr learned my addr 172.21.15.2:0/1683228698 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f906a800640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f9bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f906a800640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f9b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 -- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 msgr2=0x56408f9b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f9b6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 -- 172.21.15.2:0/1683228698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 msgr2=0x56408f9bcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f9bcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.506+0000 7f9061400640 1 -- 172.21.15.2:0/1683228698 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56408f8b3c50 con 0x56408f94eff0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f9061400640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9054011960 tx=0x7f9054004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4f10ecce847ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f904f400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054019ba0 con 0x56408f94eff0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56408f5b5600 con 0x56408f94eff0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f9054000e40 con 0x56408f94eff0 2024-03-11T18:41:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9054019d00 con 0x56408f94eff0 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f9073636740 10 client.26089 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.507+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56408f8762c0 con 0x56408f94eff0 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56408f5b5600 con 0x56408f94eff0 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9054003710 con 0x56408f94eff0 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 10 client.26089 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 10 client.26089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 20 client.26089 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f9c4710 0x56408f9c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 4 client.26089 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56408f6061b0 con 0x56408f9c4710 2024-03-11T18:41:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.508+0000 7f906a800640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f9c4710 0x56408f9c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.509+0000 7f906a800640 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f9c4710 0x56408f9c6b30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7864ab99d0eb2c41 server_cookie=a17cf1081d777726 in_seq=0 out_seq=0 2024-03-11T18:41:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.509+0000 7f904f400640 10 client.26089 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.729+0000 7f904f400640 1 -- 172.21.15.2:0/1683228698 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56408f6061b0 con 0x56408f9c4710 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f904f400640 10 client.26089 handle_command_reply: tid=0 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 client.26089 shutdown 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 msgr2=0x56408f9bd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9054011960 tx=0x7f9054004550 comp rx=0 tx=0).stop 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 shutdown_connections 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56408f9c4710 0x56408f9c6b30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56408f958570 0x56408f9b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56408f94eff0 0x56408f9bd0e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 --2- 172.21.15.2:0/1683228698 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56408f949f20 0x56408f9bcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 >> 172.21.15.2:0/1683228698 conn(0x56408f9164b0 msgr2=0x56408f95c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f904f400640 0 client.26089 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.730+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 shutdown_connections 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.731+0000 7f9073636740 1 -- 172.21.15.2:0/1683228698 wait complete. 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.731+0000 7f9073636740 20 client.26089 trim_cache size 0 max 16384 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:41:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 msgr2=0x7f90500a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 0x7f90500a40c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x56408f9504c0 tx=0x7f905c0f3c00 comp rx=0 tx=0).stop 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 msgr2=0x7f906c197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f905c000c00 tx=0x7f905c02c5f0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 shutdown_connections 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f90500a1c00 0x7f90500a40c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f906c19bee0 0x7f906c19c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.861 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f906c106370 0x7f906c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 --2- 172.21.15.2:0/1605907922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f906c068be0 0x7f906c197620 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f905c000c00 tx=0x7f905c02c5f0 comp rx=0 tx=0).stop 2024-03-11T18:41:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 >> 172.21.15.2:0/1605907922 conn(0x7f906c0759e0 msgr2=0x7f906c0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.732+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 shutdown_connections 2024-03-11T18:41:21.862 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:21.733+0000 7f906b200640 1 -- 172.21.15.2:0/1605907922 wait complete. 2024-03-11T18:41:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:21 smithi002 ceph-mon[31328]: pgmap v237: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:21 smithi165 ceph-mon[36634]: pgmap v237: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:22.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:21 smithi023 ceph-mon[36406]: pgmap v237: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:22.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 582539 7.81 MB/sec execute 262 sec latency 91.396 ms 2024-03-11T18:41:23.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 583619 7.80 MB/sec execute 263 sec latency 24.346 ms 2024-03-11T18:41:24.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:23 smithi002 ceph-mon[31328]: pgmap v238: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:24.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:23 smithi165 ceph-mon[36634]: pgmap v238: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:24.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:23 smithi023 ceph-mon[36406]: pgmap v238: 129 pgs: 129 active+clean; 96 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 186 op/s 2024-03-11T18:41:24.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 585138 7.79 MB/sec execute 264 sec latency 24.154 ms 2024-03-11T18:41:25.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 586490 7.80 MB/sec execute 265 sec latency 51.984 ms 2024-03-11T18:41:26.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:25 smithi002 ceph-mon[31328]: pgmap v239: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-03-11T18:41:26.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:25 smithi165 ceph-mon[36634]: pgmap v239: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-03-11T18:41:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:25 smithi023 ceph-mon[36406]: pgmap v239: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-03-11T18:41:26.428 INFO:tasks.workunit.client.0.smithi002.stdout: 1 587564 7.79 MB/sec execute 266 sec latency 45.201 ms 2024-03-11T18:41:27.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 589082 7.79 MB/sec execute 267 sec latency 30.658 ms 2024-03-11T18:41:28.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:27 smithi002 ceph-mon[31328]: pgmap v240: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.2 MiB/s wr, 226 op/s 2024-03-11T18:41:28.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:27 smithi165 ceph-mon[36634]: pgmap v240: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.2 MiB/s wr, 226 op/s 2024-03-11T18:41:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:27 smithi023 ceph-mon[36406]: pgmap v240: 129 pgs: 129 active+clean; 89 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.2 MiB/s wr, 226 op/s 2024-03-11T18:41:28.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 590236 7.79 MB/sec execute 268 sec latency 28.260 ms 2024-03-11T18:41:29.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 591288 7.78 MB/sec execute 269 sec latency 25.057 ms 2024-03-11T18:41:29.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:29 smithi002 ceph-mon[31328]: pgmap v241: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 266 op/s 2024-03-11T18:41:29.905 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:29 smithi165 ceph-mon[36634]: pgmap v241: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 266 op/s 2024-03-11T18:41:29.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:29 smithi023 ceph-mon[36406]: pgmap v241: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 266 op/s 2024-03-11T18:41:30.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 592946 7.78 MB/sec execute 270 sec latency 38.609 ms 2024-03-11T18:41:31.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 593843 7.77 MB/sec execute 271 sec latency 76.826 ms 2024-03-11T18:41:31.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:31 smithi165 ceph-mon[36634]: pgmap v242: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.1 MiB/s wr, 230 op/s 2024-03-11T18:41:31.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:31 smithi023 ceph-mon[36406]: pgmap v242: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.1 MiB/s wr, 230 op/s 2024-03-11T18:41:32.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:31 smithi002 ceph-mon[31328]: pgmap v242: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.1 MiB/s wr, 230 op/s 2024-03-11T18:41:32.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 594975 7.76 MB/sec execute 272 sec latency 24.320 ms 2024-03-11T18:41:33.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 596799 7.78 MB/sec execute 273 sec latency 27.058 ms 2024-03-11T18:41:33.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:33 smithi165 ceph-mon[36634]: pgmap v243: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-03-11T18:41:33.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:33 smithi023 ceph-mon[36406]: pgmap v243: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-03-11T18:41:34.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:33 smithi002 ceph-mon[31328]: pgmap v243: 129 pgs: 129 active+clean; 93 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-03-11T18:41:34.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 597691 7.76 MB/sec execute 274 sec latency 22.999 ms 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.883+0000 7f9a8a400640 1 -- 172.21.15.2:0/1953071163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a84103dc0 msgr2=0x7f9a84107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.883+0000 7f9a8a400640 1 --2- 172.21.15.2:0/1953071163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a84103dc0 0x7f9a84107e00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009900 tx=0x7f9a7002bc30 comp rx=0 tx=0).stop 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.883+0000 7f9a8a400640 1 -- 172.21.15.2:0/1953071163 shutdown_connections 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 --2- 172.21.15.2:0/1953071163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a841084e0 0x7f9a841100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 --2- 172.21.15.2:0/1953071163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a84103dc0 0x7f9a84107e00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.887 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 --2- 172.21.15.2:0/1953071163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84103410 0x7f9a841037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 -- 172.21.15.2:0/1953071163 >> 172.21.15.2:0/1953071163 conn(0x7f9a840fec90 msgr2=0x7f9a841010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 -- 172.21.15.2:0/1953071163 shutdown_connections 2024-03-11T18:41:34.888 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.884+0000 7f9a8a400640 1 -- 172.21.15.2:0/1953071163 wait complete. 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 Processor -- start 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 -- start start 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a84103410 0x7f9a8419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84199650 0x7f9a84199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.889 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419a120 con 0x7f9a84103410 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419a290 con 0x7f9a84199650 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a8a400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8419a450 con 0x7f9a841084e0 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a89000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a84103410 0x7f9a8419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:39578/0 (socket says 172.21.15.2:39578) 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a83e00640 1 -- 172.21.15.2:0/3983176991 learned_addr learned my addr 172.21.15.2:0/3983176991 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.885+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84199650 0x7f9a84199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 -- 172.21.15.2:0/3983176991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84199650 msgr2=0x7f9a84199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84199650 0x7f9a84199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 -- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a84103410 msgr2=0x7f9a8419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a84103410 0x7f9a8419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.890 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 -- 172.21.15.2:0/3983176991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a70009560 con 0x7f9a841084e0 2024-03-11T18:41:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.886+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a70000c00 tx=0x7f9a7002c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.887+0000 7f9a81600640 1 -- 172.21.15.2:0/3983176991 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7002dea0 con 0x7f9a841084e0 2024-03-11T18:41:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.887+0000 7f9a81600640 1 -- 172.21.15.2:0/3983176991 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a700042d0 con 0x7f9a841084e0 2024-03-11T18:41:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.887+0000 7f9a81600640 1 -- 172.21.15.2:0/3983176991 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70003a90 con 0x7f9a841084e0 2024-03-11T18:41:34.891 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.887+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a841a6390 con 0x7f9a841084e0 2024-03-11T18:41:34.892 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.887+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a841a68c0 con 0x7f9a841084e0 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.890+0000 7f9a81600640 1 -- 172.21.15.2:0/3983176991 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f9a700053d0 con 0x7f9a841084e0 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.890+0000 7f9a8e639740 1 Processor -- start 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.890+0000 7f9a8e639740 1 -- start start 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd46ff0 0x559f0bd47470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd52960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bd02620 con 0x559f0bd46ff0 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bcb48c0 con 0x559f0bd47d60 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a8e639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bcb1df0 con 0x559f0bd50570 2024-03-11T18:41:34.894 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60148/0 (socket says 172.21.15.2:60148) 2024-03-11T18:41:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd52960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 -- 172.21.15.2:0/3540236411 learned_addr learned my addr 172.21.15.2:0/3540236411 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89000640 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd46ff0 0x559f0bd47470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 -- 172.21.15.2:0/3540236411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 msgr2=0x559f0bd52960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd52960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 -- 172.21.15.2:0/3540236411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd46ff0 msgr2=0x559f0bd47470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd46ff0 0x559f0bd47470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.891+0000 7f9a89a00640 1 -- 172.21.15.2:0/3540236411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f0bc2eb00 con 0x559f0bd47d60 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a81600640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 0x7f9a6c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009f50 tx=0x7f9a7c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f63c2d23fbd8fe server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a81600640 1 -- 172.21.15.2:0/3983176991 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f9a700f0070 con 0x7f9a841084e0 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a76000640 1 -- 172.21.15.2:0/3540236411 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c004d20 con 0x559f0bd47d60 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a76000640 1 -- 172.21.15.2:0/3540236411 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a7c004e80 con 0x559f0bd47d60 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a76000640 1 -- 172.21.15.2:0/3540236411 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c005280 con 0x559f0bd47d60 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.892+0000 7f9a89000640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 0x7f9a6c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 -- 172.21.15.2:0/3540236411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 msgr2=0x559f0bd46ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009f50 tx=0x7f9a7c0118d0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a89000640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 0x7f9a6c0a40c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x559f0bd484c0 tx=0x7f9a78005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 -- 172.21.15.2:0/3540236411 shutdown_connections 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd52960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd46ff0 0x559f0bd47470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3540236411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd47d60 0x559f0bd46ab0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 -- 172.21.15.2:0/3540236411 >> 172.21.15.2:0/3540236411 conn(0x559f0bd0e4b0 msgr2=0x559f0bd41ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 -- 172.21.15.2:0/3540236411 shutdown_connections 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.893+0000 7f9a8e639740 1 -- 172.21.15.2:0/3540236411 wait complete. 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 -- start start 2024-03-11T18:41:34.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 Processor -- start 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 0x559f0bd4db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 0x559f0bd4e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bcad5d0 con 0x559f0bd50570 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bc52b50 con 0x559f0bd46ff0 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.894+0000 7f9a8e639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bc527f0 con 0x559f0bd47d60 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a8e639740 10 client.0 _ll_register_callbacks cb 0x559f0bc66550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60160/0 (socket says 172.21.15.2:60160) 2024-03-11T18:41:34.902 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 -- 172.21.15.2:0/3535108648 learned_addr learned my addr 172.21.15.2:0/3535108648 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 0x559f0bd4e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 0x559f0bd4db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 -- 172.21.15.2:0/3535108648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 msgr2=0x559f0bd4db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 0x559f0bd4db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 -- 172.21.15.2:0/3535108648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 msgr2=0x559f0bd4e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 0x559f0bd4e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 -- 172.21.15.2:0/3535108648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c009c00 con 0x559f0bd46ff0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 0x559f0bd4e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 0x559f0bd4db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a89000640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009730 tx=0x7f9a78004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a32b6bca649328 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.895+0000 7f9a6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78017070 con 0x559f0bd46ff0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f9a7801f640 con 0x559f0bd46ff0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f0bc7fcd0 con 0x559f0bd46ff0 2024-03-11T18:41:34.903 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a7801f800 con 0x559f0bd46ff0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a8e639740 10 client.26108 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559f0bcabc50 con 0x559f0bd46ff0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559f0bc7fcd0 con 0x559f0bd46ff0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.896+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9a78029ad0 con 0x559f0bd46ff0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 10 client.26108 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 20 client.26108 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bd60680 0x559f0bd62aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 4 client.26108 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f0bca3ce0 con 0x559f0bd60680 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bd60680 0x559f0bd62aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a83e00640 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bd60680 0x559f0bd62aa0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266e0a90eb37eba2 server_cookie=de75d2a118dafbdd in_seq=0 out_seq=0 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.897+0000 7f9a6aa00640 10 client.26108 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.898+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3535108648 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559f0bca3ce0 con 0x559f0bd60680 2024-03-11T18:41:34.904 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.898+0000 7f9a6aa00640 10 client.26108 handle_command_reply: tid=0 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.898+0000 7f9a8e639740 1 client.26108 shutdown 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.898+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 msgr2=0x559f0bd4d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.898+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009730 tx=0x7f9a78004730 comp rx=0 tx=0).stop 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 shutdown_connections 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bd60680 0x559f0bd62aa0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd50570 0x559f0bd4e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd47d60 0x559f0bd4db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3535108648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4d610 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a6aa00640 0 client.26108 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 >> 172.21.15.2:0/3535108648 conn(0x559f0bd0e4b0 msgr2=0x559f0bd43a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 shutdown_connections 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 1 -- 172.21.15.2:0/3535108648 wait complete. 2024-03-11T18:41:34.905 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.899+0000 7f9a8e639740 20 client.26108 trim_cache size 0 max 16384 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 Processor -- start 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 -- start start 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd4deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd4a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0b9f5e60 con 0x559f0bd47d60 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bd52dc0 con 0x559f0bd46ff0 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a8e639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bca3ce0 con 0x559f0bd50570 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a89a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd4a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60168/0 (socket says 172.21.15.2:60168) 2024-03-11T18:41:34.920 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a89000640 1 -- 172.21.15.2:0/1454385557 learned_addr learned my addr 172.21.15.2:0/1454385557 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.916+0000 7f9a83e00640 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd4deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 -- 172.21.15.2:0/1454385557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 msgr2=0x559f0bd4a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd4a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 -- 172.21.15.2:0/1454385557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 msgr2=0x559f0bd4deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd4deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 -- 172.21.15.2:0/1454385557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f0bc48530 con 0x559f0bd46ff0 2024-03-11T18:41:34.921 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a83e00640 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd4deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a89000640 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a78012d50 tx=0x7f9a78003e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17071340c6e62d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a75600640 1 -- 172.21.15.2:0/1454385557 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78005220 con 0x559f0bd46ff0 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a75600640 1 -- 172.21.15.2:0/1454385557 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a78004c80 con 0x559f0bd46ff0 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.917+0000 7f9a75600640 1 -- 172.21.15.2:0/1454385557 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7801f9b0 con 0x559f0bd46ff0 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 -- 172.21.15.2:0/1454385557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 msgr2=0x559f0bd4fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9a78012d50 tx=0x7f9a78003e50 comp rx=0 tx=0).stop 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 -- 172.21.15.2:0/1454385557 shutdown_connections 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd4a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd4deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 --2- 172.21.15.2:0/1454385557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd4fd40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.918+0000 7f9a8e639740 1 -- 172.21.15.2:0/1454385557 >> 172.21.15.2:0/1454385557 conn(0x559f0bd0e4b0 msgr2=0x559f0bd4bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.919+0000 7f9a8e639740 1 -- 172.21.15.2:0/1454385557 shutdown_connections 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.919+0000 7f9a8e639740 1 -- 172.21.15.2:0/1454385557 wait complete. 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.919+0000 7f9a8e639740 1 -- start start 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.919+0000 7f9a8e639740 1 Processor -- start 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd56220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd56760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bd57c80 con 0x559f0bd47d60 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bdbf3e0 con 0x559f0bd46ff0 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0bdbf570 con 0x559f0bd50570 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a8e639740 10 client.0 _ll_register_callbacks cb 0x7f9a78009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd56760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60180/0 (socket says 172.21.15.2:60180) 2024-03-11T18:41:34.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd56220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.925 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89000640 1 -- 172.21.15.2:0/3467673386 learned_addr learned my addr 172.21.15.2:0/3467673386 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 -- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 msgr2=0x559f0bd55ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 -- 172.21.15.2:0/3467673386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 msgr2=0x559f0bd56220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd56220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89a00640 1 -- 172.21.15.2:0/3467673386 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f0bcabc50 con 0x559f0bd50570 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.920+0000 7f9a89000640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a89a00640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd56760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011dd0 tx=0x7f9a7c005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66ada34e88c411c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c005730 con 0x559f0bd50570 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f0bcd2d30 con 0x559f0bd50570 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f9a7c02a070 con 0x559f0bd50570 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a7c005bd0 con 0x559f0bd50570 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a8e639740 10 client.26107 fetch_fsmap learned FSMap version 29 2024-03-11T18:41:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.921+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559f0bc6e2c0 con 0x559f0bd50570 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559f0bcd2d30 con 0x559f0bd50570 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9a7c02e450 con 0x559f0bd50570 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 10 client.26107 fetch_fsmap finished waiting for FSMap version 29 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 10 client.26107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 20 client.26107 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bdc0d20 0x559f0bdc3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 4 client.26107 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559f0b9ad600 con 0x559f0bdc0d20 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.922+0000 7f9a89000640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bdc0d20 0x559f0bdc3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.923+0000 7f9a89000640 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bdc0d20 0x559f0bdc3140 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22ebdde0cab8ab83 server_cookie=73cd7dfc20e01ddf in_seq=0 out_seq=0 2024-03-11T18:41:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:34.923+0000 7f9a6aa00640 10 client.26107 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.166+0000 7f9a6aa00640 1 -- 172.21.15.2:0/3467673386 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559f0b9ad600 con 0x559f0bdc0d20 2024-03-11T18:41:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.166+0000 7f9a6aa00640 10 client.26107 handle_command_reply: tid=0 2024-03-11T18:41:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.166+0000 7f9a8e639740 1 client.26107 shutdown 2024-03-11T18:41:35.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.166+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 msgr2=0x559f0bd56760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.166+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd56760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011dd0 tx=0x7f9a7c005230 comp rx=0 tx=0).stop 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 shutdown_connections 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559f0bdc0d20 0x559f0bdc3140 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559f0bd50570 0x559f0bd56760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559f0bd47d60 0x559f0bd56220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 --2- 172.21.15.2:0/3467673386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559f0bd46ff0 0x559f0bd55ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 >> 172.21.15.2:0/3467673386 conn(0x559f0bd0e4b0 msgr2=0x559f0bd404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a6aa00640 0 client.26107 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.167+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 shutdown_connections 2024-03-11T18:41:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.168+0000 7f9a8e639740 1 -- 172.21.15.2:0/3467673386 wait complete. 2024-03-11T18:41:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.168+0000 7f9a8e639740 20 client.26107 trim_cache size 0 max 16384 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 msgr2=0x7f9a6c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 0x7f9a6c0a40c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x559f0bd484c0 tx=0x7f9a78005f90 comp rx=0 tx=0).stop 2024-03-11T18:41:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 msgr2=0x7f9a8419fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a70000c00 tx=0x7f9a7002c670 comp rx=0 tx=0).stop 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 shutdown_connections 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a84199650 0x7f9a84199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a841084e0 0x7f9a8419fad0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f9a6c0a1c00 0x7f9a6c0a40c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 --2- 172.21.15.2:0/3983176991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a84103410 0x7f9a8419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:35.173 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.170+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 >> 172.21.15.2:0/3983176991 conn(0x7f9a840fec90 msgr2=0x7f9a84100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.171+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 shutdown_connections 2024-03-11T18:41:35.174 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:35.171+0000 7f9a8a400640 1 -- 172.21.15.2:0/3983176991 wait complete. 2024-03-11T18:41:35.429 INFO:tasks.workunit.client.0.smithi002.stdout: 1 599078 7.75 MB/sec execute 275 sec latency 110.401 ms 2024-03-11T18:41:35.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:35 smithi165 ceph-mon[36634]: pgmap v244: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 292 op/s 2024-03-11T18:41:35.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:41:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setuser ceph since I am not root 2024-03-11T18:41:35.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:41:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setgroup ceph since I am not root 2024-03-11T18:41:35.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:41:35 smithi165 ceph-mds[67373]: -- 172.21.15.165:0/1842490958 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a9982a81a0 con 0x55a99827ec00 2024-03-11T18:41:35.906 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:41:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: starting mds.c at 2024-03-11T18:41:35.907 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:35 smithi023 ceph-mon[36406]: pgmap v244: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 292 op/s 2024-03-11T18:41:36.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:35 smithi002 ceph-mon[31328]: pgmap v244: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 292 op/s 2024-03-11T18:41:36.430 INFO:tasks.workunit.client.0.smithi002.stdout: 1 600541 7.77 MB/sec execute 276 sec latency 57.862 ms 2024-03-11T18:41:37.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:36 smithi002 ceph-mon[31328]: daemon mds.c restarted 2024-03-11T18:41:37.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:36 smithi165 ceph-mon[36634]: daemon mds.c restarted 2024-03-11T18:41:37.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:36 smithi023 ceph-mon[36406]: daemon mds.c restarted 2024-03-11T18:41:37.430 INFO:tasks.workunit.client.0.smithi002.stdout: 1 601506 7.75 MB/sec execute 277 sec latency 25.327 ms 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: pgmap v245: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6836/1442654134,v1:172.21.15.165:6837/1442654134] up:boot 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:41:38.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:37 smithi002 ceph-mon[31328]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: pgmap v245: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6836/1442654134,v1:172.21.15.165:6837/1442654134] up:boot 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:37 smithi165 ceph-mon[36634]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: pgmap v245: 129 pgs: 129 active+clean; 97 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 142 op/s 2024-03-11T18:41:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: osdmap e102: 12 total, 12 up, 12 in 2024-03-11T18:41:38.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:41:38.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6836/1442654134,v1:172.21.15.165:6837/1442654134] up:boot 2024-03-11T18:41:38.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-11T18:41:38.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:37 smithi023 ceph-mon[36406]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-11T18:41:38.430 INFO:tasks.workunit.client.0.smithi002.stdout: 1 602862 7.74 MB/sec execute 278 sec latency 25.983 ms 2024-03-11T18:41:39.430 INFO:tasks.workunit.client.0.smithi002.stdout: 1 604315 7.75 MB/sec execute 279 sec latency 14.210 ms 2024-03-11T18:41:40.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:39 smithi002 ceph-mon[31328]: pgmap v247: 129 pgs: 129 active+clean; 94 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 278 op/s 2024-03-11T18:41:40.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:39 smithi165 ceph-mon[36634]: pgmap v247: 129 pgs: 129 active+clean; 94 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 278 op/s 2024-03-11T18:41:40.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:39 smithi023 ceph-mon[36406]: pgmap v247: 129 pgs: 129 active+clean; 94 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 278 op/s 2024-03-11T18:41:40.430 INFO:tasks.workunit.client.0.smithi002.stdout: 1 605452 7.74 MB/sec execute 280 sec latency 23.531 ms 2024-03-11T18:41:41.431 INFO:tasks.workunit.client.0.smithi002.stdout: 1 607312 7.76 MB/sec execute 281 sec latency 54.825 ms 2024-03-11T18:41:42.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:41 smithi002 ceph-mon[31328]: pgmap v248: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:41 smithi165 ceph-mon[36634]: pgmap v248: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:41 smithi023 ceph-mon[36406]: pgmap v248: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:42.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 608323 7.74 MB/sec execute 282 sec latency 22.802 ms 2024-03-11T18:41:43.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 609965 7.74 MB/sec execute 283 sec latency 22.943 ms 2024-03-11T18:41:44.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:43 smithi002 ceph-mon[31328]: pgmap v249: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:44.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:43 smithi165 ceph-mon[36634]: pgmap v249: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:44.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:43 smithi023 ceph-mon[36406]: pgmap v249: 129 pgs: 129 active+clean; 90 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 270 op/s 2024-03-11T18:41:44.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 611565 7.75 MB/sec execute 284 sec latency 30.068 ms 2024-03-11T18:41:45.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 612755 7.74 MB/sec execute 285 sec latency 29.256 ms 2024-03-11T18:41:46.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:45 smithi002 ceph-mon[31328]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:45 smithi165 ceph-mon[36634]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:45 smithi023 ceph-mon[36406]: pgmap v250: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:46.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 614443 7.76 MB/sec execute 286 sec latency 47.106 ms 2024-03-11T18:41:47.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 615320 7.74 MB/sec execute 287 sec latency 30.505 ms 2024-03-11T18:41:47.957 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:47 smithi165 ceph-mon[36634]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:48.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:47 smithi002 ceph-mon[31328]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:47 smithi023 ceph-mon[36406]: pgmap v251: 129 pgs: 129 active+clean; 95 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 268 op/s 2024-03-11T18:41:48.325 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 -- 172.21.15.2:0/2944179870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c50103f90 msgr2=0x7f7c50107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 --2- 172.21.15.2:0/2944179870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c50103f90 0x7f7c50107fd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009940 tx=0x7f7c3c02bc20 comp rx=0 tx=0).stop 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 -- 172.21.15.2:0/2944179870 shutdown_connections 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 --2- 172.21.15.2:0/2944179870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c50110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 --2- 172.21.15.2:0/2944179870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c50103f90 0x7f7c50107fd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 --2- 172.21.15.2:0/2944179870 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c501035e0 0x7f7c501039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 -- 172.21.15.2:0/2944179870 >> 172.21.15.2:0/2944179870 conn(0x7f7c500fee00 msgr2=0x7f7c50101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 -- 172.21.15.2:0/2944179870 shutdown_connections 2024-03-11T18:41:48.326 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.323+0000 7f7c55800640 1 -- 172.21.15.2:0/2944179870 wait complete. 2024-03-11T18:41:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.324+0000 7f7c55800640 1 Processor -- start 2024-03-11T18:41:48.327 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 -- start start 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c5019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c50199850 0x7f7c50199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5019a300 con 0x7f7c50199850 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5019a470 con 0x7f7c501035e0 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c55800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5019a5e0 con 0x7f7c501086b0 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.328 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.325+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c5019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55816/0 (socket says 172.21.15.2:55816) 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c5019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40470/0 (socket says 172.21.15.2:40470) 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2682335399 learned_addr learned my addr 172.21.15.2:0/2682335399 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4f400640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c50199850 0x7f7c50199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2682335399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 msgr2=0x7f7c5019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c5019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c50199850 msgr2=0x7f7c50199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c50199850 0x7f7c50199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2682335399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c009500 con 0x7f7c501035e0 2024-03-11T18:41:48.329 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.326+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000e9f0 tx=0x7f7c4000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.327+0000 7f7c47400640 1 -- 172.21.15.2:0/2682335399 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40009700 con 0x7f7c501035e0 2024-03-11T18:41:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.327+0000 7f7c47400640 1 -- 172.21.15.2:0/2682335399 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c4000cd80 con 0x7f7c501035e0 2024-03-11T18:41:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.327+0000 7f7c47400640 1 -- 172.21.15.2:0/2682335399 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40018920 con 0x7f7c501035e0 2024-03-11T18:41:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.327+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c501a6590 con 0x7f7c501035e0 2024-03-11T18:41:48.330 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.327+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c501a6a20 con 0x7f7c501035e0 2024-03-11T18:41:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.331+0000 7f7c47400640 1 -- 172.21.15.2:0/2682335399 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f7c40018a80 con 0x7f7c501035e0 2024-03-11T18:41:48.333 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.331+0000 7f7c58429740 1 Processor -- start 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 -- start start 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f369018870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369019230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368fd4620 con 0x55f369019b20 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368f868c0 con 0x55f36902ab40 2024-03-11T18:41:48.335 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c58429740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368f83df0 con 0x55f369018db0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369019230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55828/0 (socket says 172.21.15.2:55828) 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c4e000640 1 -- 172.21.15.2:0/1871854818 learned_addr learned my addr 172.21.15.2:0/1871854818 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.332+0000 7f7c4f400640 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f369018870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 -- 172.21.15.2:0/1871854818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 msgr2=0x55f369019230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369019230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c47400640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 0x7f7c340a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 -- 172.21.15.2:0/1871854818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 msgr2=0x55f369018870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f369018870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 -- 172.21.15.2:0/1871854818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f368f00b00 con 0x55f36902ab40 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c47400640 1 -- 172.21.15.2:0/2682335399 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7c40014070 con 0x7f7c501035e0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4f400640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 0x7f7c340a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.336 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.333+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c02c230 tx=0x7f7c3c005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e7ec7afb0d529cf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c3be00640 1 -- 172.21.15.2:0/1871854818 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c004240 con 0x55f36902ab40 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c3be00640 1 -- 172.21.15.2:0/1871854818 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c3c005510 con 0x55f36902ab40 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c3be00640 1 -- 172.21.15.2:0/1871854818 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c02c5c0 con 0x55f36902ab40 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c58429740 1 -- 172.21.15.2:0/1871854818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 msgr2=0x55f36902cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c4f400640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 0x7f7c340a40d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005c40 tx=0x7f7c48004a40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.334+0000 7f7c58429740 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c02c230 tx=0x7f7c3c005d70 comp rx=0 tx=0).stop 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 -- 172.21.15.2:0/1871854818 shutdown_connections 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f36902cf30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369019230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 --2- 172.21.15.2:0/1871854818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f369018870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.341 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 -- 172.21.15.2:0/1871854818 >> 172.21.15.2:0/1871854818 conn(0x55f368fe04b0 msgr2=0x55f369013aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 -- 172.21.15.2:0/1871854818 shutdown_connections 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 -- 172.21.15.2:0/1871854818 wait complete. 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.335+0000 7f7c58429740 1 -- start start 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 Processor -- start 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f3690280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369028600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368f7f5d0 con 0x55f369019b20 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368f24b50 con 0x55f36902ab40 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368f247f0 con 0x55f369018db0 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c58429740 10 client.0 _ll_register_callbacks cb 0x55f368f38550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.342 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c4f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369028600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40492/0 (socket says 172.21.15.2:40492) 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.336+0000 7f7c4ea00640 1 -- 172.21.15.2:0/646740613 learned_addr learned my addr 172.21.15.2:0/646740613 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4e000640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f3690280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 -- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 msgr2=0x55f369028600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369028600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 -- 172.21.15.2:0/646740613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 msgr2=0x55f3690280c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f3690280c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 -- 172.21.15.2:0/646740613 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c50110a90 con 0x55f369018db0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4f400640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369028600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c6a0 tx=0x7f7c4009a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdda3d158e5aba4b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c38c00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c400b8ea0 con 0x55f369018db0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7c4009a7e0 con 0x55f369018db0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.337+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f368f51cd0 con 0x55f369018db0 2024-03-11T18:41:48.343 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c4009e880 con 0x55f369018db0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 10 client.26125 fetch_fsmap learned FSMap version 31 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f368f7dc50 con 0x55f369018db0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c400b2070 con 0x55f369018db0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7c400ba100 con 0x55f369018db0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 10 client.26125 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 10 client.26125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 20 client.26125 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f36903ab60 0x55f36903cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 4 client.26125 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f368f75ce0 con 0x55f36903ab60 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.338+0000 7f7c4e000640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f36903ab60 0x55f36903cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.339+0000 7f7c4e000640 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f36903ab60 0x55f36903cfa0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdd832ca227f7006 server_cookie=9244955fe045af54 in_seq=0 out_seq=0 2024-03-11T18:41:48.344 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.339+0000 7f7c38c00640 10 client.26125 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c38c00640 1 -- 172.21.15.2:0/646740613 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f368f75ce0 con 0x55f36903ab60 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c38c00640 10 client.26125 handle_command_reply: tid=0 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 client.26125 shutdown 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 msgr2=0x55f369027b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c6a0 tx=0x7f7c4009a5b0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 shutdown_connections 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f36903ab60 0x55f36903cfa0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369028600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369019b20 0x55f3690280c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 --2- 172.21.15.2:0/646740613 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369018db0 0x55f369027b80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c38c00640 0 client.26125 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 >> 172.21.15.2:0/646740613 conn(0x55f368fe04b0 msgr2=0x55f3690157f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.346 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.340+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 shutdown_connections 2024-03-11T18:41:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.341+0000 7f7c58429740 1 -- 172.21.15.2:0/646740613 wait complete. 2024-03-11T18:41:48.347 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.341+0000 7f7c58429740 20 client.26125 trim_cache size 0 max 16384 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 Processor -- start 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 -- start start 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369018db0 0x55f36902a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369031e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f368cc7e60 con 0x55f369018db0 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f369015df0 con 0x55f36902ab40 2024-03-11T18:41:48.359 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c58429740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f36900f3e0 con 0x55f369019b20 2024-03-11T18:41:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.360 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:40496/0 (socket says 172.21.15.2:40496) 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c4e000640 1 -- 172.21.15.2:0/1468886955 learned_addr learned my addr 172.21.15.2:0/1468886955 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.357+0000 7f7c4f400640 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369031e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369018db0 0x55f36902a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 -- 172.21.15.2:0/1468886955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 msgr2=0x55f369031e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369031e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 -- 172.21.15.2:0/1468886955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369018db0 msgr2=0x55f36902a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369018db0 0x55f36902a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 -- 172.21.15.2:0/1468886955 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f368f1a530 con 0x55f369019b20 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c4e000640 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009860 tx=0x7f7c3c041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d5476801333c2b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c3b400640 1 -- 172.21.15.2:0/1468886955 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c034a00 con 0x55f369019b20 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c3b400640 1 -- 172.21.15.2:0/1468886955 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c3c005040 con 0x55f369019b20 2024-03-11T18:41:48.361 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.358+0000 7f7c3b400640 1 -- 172.21.15.2:0/1468886955 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c04e5d0 con 0x55f369019b20 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 -- 172.21.15.2:0/1468886955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 msgr2=0x55f369031900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009860 tx=0x7f7c3c041040 comp rx=0 tx=0).stop 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 -- 172.21.15.2:0/1468886955 shutdown_connections 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f36902ab40 0x55f369031e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f369031900 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 --2- 172.21.15.2:0/1468886955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f369018db0 0x55f36902a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 -- 172.21.15.2:0/1468886955 >> 172.21.15.2:0/1468886955 conn(0x55f368fe04b0 msgr2=0x55f3690135d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 -- 172.21.15.2:0/1468886955 shutdown_connections 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.359+0000 7f7c58429740 1 -- 172.21.15.2:0/1468886955 wait complete. 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 -- start start 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 Processor -- start 2024-03-11T18:41:48.365 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f36902dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f36902ab40 0x55f36902e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f36902ec00 con 0x55f36902ab40 2024-03-11T18:41:48.366 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f36902ed70 con 0x55f369018db0 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3690910c0 con 0x55f369019b20 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.360+0000 7f7c58429740 10 client.0 _ll_register_callbacks cb 0x7f7c400d14a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4e000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f36902dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55856/0 (socket says 172.21.15.2:55856) 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2839793218 learned_addr learned my addr 172.21.15.2:0/2839793218 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:41:48.367 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4f400640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f36902ab40 0x55f36902e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2839793218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 msgr2=0x55f36902dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f36902dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2839793218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f36902ab40 msgr2=0x55f36902e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f36902ab40 0x55f36902e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 -- 172.21.15.2:0/2839793218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f368f00b00 con 0x55f369018db0 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c4ea00640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7c400a29a0 tx=0x7f7c400d1550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fd2e21abcaaa61b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:41:48.368 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c38c00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c400b8ea0 con 0x55f369018db0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f368fa4d30 con 0x55f369018db0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.361+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f7c400b3230 con 0x55f369018db0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c4009a320 con 0x55f369018db0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 10 client.26138 fetch_fsmap learned FSMap version 31 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f368f402c0 con 0x55f369018db0 2024-03-11T18:41:48.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f368fa4d30 con 0x55f369018db0 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7c400cb4d0 con 0x55f369018db0 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 10 client.26138 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 20 client.26138 populate_metadata read hostname 'smithi002' 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f369092480 0x55f369094870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 4 client.26138 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:41:48.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f368c7f600 con 0x55f369092480 2024-03-11T18:41:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.362+0000 7f7c4e000640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f369092480 0x55f369094870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:41:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.363+0000 7f7c4e000640 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f369092480 0x55f369094870 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13acc02816445965 server_cookie=d9274eede2c40a08 in_seq=0 out_seq=0 2024-03-11T18:41:48.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.363+0000 7f7c38c00640 10 client.26138 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:48.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 616529 7.73 MB/sec execute 288 sec latency 66.763 ms 2024-03-11T18:41:48.598 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c38c00640 1 -- 172.21.15.2:0/2839793218 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f368c7f600 con 0x55f369092480 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c38c00640 10 client.26138 handle_command_reply: tid=0 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 client.26138 shutdown 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 msgr2=0x55f369031c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7c400a29a0 tx=0x7f7c400d1550 comp rx=0 tx=0).stop 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 shutdown_connections 2024-03-11T18:41:48.599 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55f369092480 0x55f369094870 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55f36902ab40 0x55f36902e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55f369019b20 0x55f36902dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 --2- 172.21.15.2:0/2839793218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f369018db0 0x55f369031c40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c38c00640 0 client.26138 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:41:48.600 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.595+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 >> 172.21.15.2:0/2839793218 conn(0x55f368fe04b0 msgr2=0x55f369014de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.596+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 shutdown_connections 2024-03-11T18:41:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.596+0000 7f7c58429740 1 -- 172.21.15.2:0/2839793218 wait complete. 2024-03-11T18:41:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.596+0000 7f7c58429740 20 client.26138 trim_cache size 0 max 16384 2024-03-11T18:41:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:41:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:41:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:41:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:41:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 msgr2=0x7f7c340a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 0x7f7c340a40d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005c40 tx=0x7f7c48004a40 comp rx=0 tx=0).stop 2024-03-11T18:41:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 msgr2=0x7f7c5019f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:41:48.603 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000e9f0 tx=0x7f7c4000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 shutdown_connections 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7c340a1c10 0x7f7c340a40d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c50199850 0x7f7c50199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c501086b0 0x7f7c5019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 --2- 172.21.15.2:0/2682335399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c501035e0 0x7f7c5019f6e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:41:48.604 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 >> 172.21.15.2:0/2682335399 conn(0x7f7c500fee00 msgr2=0x7f7c50100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:41:48.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 shutdown_connections 2024-03-11T18:41:48.605 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:41:48.597+0000 7f7c55800640 1 -- 172.21.15.2:0/2682335399 wait complete. 2024-03-11T18:41:49.432 INFO:tasks.workunit.client.0.smithi002.stdout: 1 618114 7.74 MB/sec execute 289 sec latency 134.711 ms 2024-03-11T18:41:50.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:49 smithi002 ceph-mon[31328]: pgmap v252: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-03-11T18:41:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:49 smithi165 ceph-mon[36634]: pgmap v252: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-03-11T18:41:50.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:49 smithi023 ceph-mon[36406]: pgmap v252: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 323 op/s 2024-03-11T18:41:50.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 619051 7.73 MB/sec execute 290 sec latency 26.182 ms 2024-03-11T18:41:51.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 620391 7.72 MB/sec execute 291 sec latency 75.721 ms 2024-03-11T18:41:52.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:51 smithi002 ceph-mon[31328]: pgmap v253: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-03-11T18:41:52.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:51 smithi165 ceph-mon[36634]: pgmap v253: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-03-11T18:41:52.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:51 smithi023 ceph-mon[36406]: pgmap v253: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 216 op/s 2024-03-11T18:41:52.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 622065 7.73 MB/sec execute 292 sec latency 15.500 ms 2024-03-11T18:41:53.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 623186 7.73 MB/sec execute 293 sec latency 24.761 ms 2024-03-11T18:41:54.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:53 smithi002 ceph-mon[31328]: pgmap v254: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:41:54.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:53 smithi165 ceph-mon[36634]: pgmap v254: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:41:54.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:53 smithi023 ceph-mon[36406]: pgmap v254: 129 pgs: 129 active+clean; 99 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:41:54.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 625089 7.74 MB/sec execute 294 sec latency 27.056 ms 2024-03-11T18:41:55.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 626066 7.72 MB/sec execute 295 sec latency 30.995 ms 2024-03-11T18:41:56.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:55 smithi002 ceph-mon[31328]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:41:56.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:55 smithi165 ceph-mon[36634]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:41:56.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:55 smithi023 ceph-mon[36406]: pgmap v255: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:41:56.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 627431 7.72 MB/sec execute 296 sec latency 101.794 ms 2024-03-11T18:41:57.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 628987 7.73 MB/sec execute 297 sec latency 21.179 ms 2024-03-11T18:41:58.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:57 smithi002 ceph-mon[31328]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:41:58.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:57 smithi165 ceph-mon[36634]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:41:58.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:57 smithi023 ceph-mon[36406]: pgmap v256: 129 pgs: 129 active+clean; 95 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-11T18:41:58.433 INFO:tasks.workunit.client.0.smithi002.stdout: 1 629876 7.71 MB/sec execute 298 sec latency 30.398 ms 2024-03-11T18:41:59.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 631400 7.72 MB/sec execute 299 sec latency 23.112 ms 2024-03-11T18:42:00.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:41:59 smithi002 ceph-mon[31328]: pgmap v257: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:42:00.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:41:59 smithi165 ceph-mon[36634]: pgmap v257: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:42:00.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:41:59 smithi023 ceph-mon[36406]: pgmap v257: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.0 MiB/s wr, 311 op/s 2024-03-11T18:42:00.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 632718 7.72 MB/sec execute 300 sec latency 16.413 ms 2024-03-11T18:42:01.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 633182 7.70 MB/sec execute 301 sec latency 197.469 ms 2024-03-11T18:42:01.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 -- 172.21.15.2:0/2877742485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 msgr2=0x7f84c01039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 --2- 172.21.15.2:0/2877742485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c01039b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009930 tx=0x7f84b002bc20 comp rx=0 tx=0).stop 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 -- 172.21.15.2:0/2877742485 shutdown_connections 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 --2- 172.21.15.2:0/2877742485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01086a0 0x7f84c0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 --2- 172.21.15.2:0/2877742485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c0103f80 0x7f84c0107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 --2- 172.21.15.2:0/2877742485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c01039b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 -- 172.21.15.2:0/2877742485 >> 172.21.15.2:0/2877742485 conn(0x7f84c00fee70 msgr2=0x7f84c0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.752+0000 7f84c6e00640 1 -- 172.21.15.2:0/2877742485 shutdown_connections 2024-03-11T18:42:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.753+0000 7f84c6e00640 1 -- 172.21.15.2:0/2877742485 wait complete. 2024-03-11T18:42:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 Processor -- start 2024-03-11T18:42:01.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 -- start start 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01998b0 0x7f84c0199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c019a2a0 con 0x7f84c01998b0 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c019a410 con 0x7f84c01035d0 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.754+0000 7f84c6e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c019a5d0 con 0x7f84c01086a0 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:43074/0 (socket says 172.21.15.2:43074) 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 -- 172.21.15.2:0/4282696715 learned_addr learned my addr 172.21.15.2:0/4282696715 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:01.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bfe00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84c4e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01998b0 0x7f84c0199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 -- 172.21.15.2:0/4282696715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 msgr2=0x7f84c019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 -- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01998b0 msgr2=0x7f84c0199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01998b0 0x7f84c0199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 -- 172.21.15.2:0/4282696715 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0009560 con 0x7f84c01086a0 2024-03-11T18:42:01.758 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bfe00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.755+0000 7f84bf400640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00e9f0 tx=0x7f84ac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.756+0000 7f84bcc00640 1 -- 172.21.15.2:0/4282696715 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0096c0 con 0x7f84c01086a0 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.756+0000 7f84bcc00640 1 -- 172.21.15.2:0/4282696715 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84ac005500 con 0x7f84c01086a0 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.756+0000 7f84bcc00640 1 -- 172.21.15.2:0/4282696715 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac010430 con 0x7f84c01086a0 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.756+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c01a6570 con 0x7f84c01086a0 2024-03-11T18:42:01.759 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.756+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c01a6a00 con 0x7f84c01086a0 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.760+0000 7f84c9b76740 1 Processor -- start 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.760+0000 7f84c9b76740 1 -- start start 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afadf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaea720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afa9a620 con 0x55b3afadedb0 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afa4c8c0 con 0x55b3afadfb20 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c9b76740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afa49df0 con 0x55b3afae8330 2024-03-11T18:42:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afadf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:46824/0 (socket says 172.21.15.2:46824) 2024-03-11T18:42:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 -- 172.21.15.2:0/1760564962 learned_addr learned my addr 172.21.15.2:0/1760564962 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:01.764 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84bcc00640 1 -- 172.21.15.2:0/4282696715 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f84ac005020 con 0x7f84c01086a0 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 -- 172.21.15.2:0/1760564962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 msgr2=0x55b3afaea720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaea720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.761+0000 7f84c4e00640 1 -- 172.21.15.2:0/1760564962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 msgr2=0x55b3afadf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84c4e00640 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afadf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84c4e00640 1 -- 172.21.15.2:0/1760564962 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3af9c6b00 con 0x55b3afadfb20 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84c4e00640 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009f50 tx=0x7f84b80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8fd9ce4e09be7da server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84b5600640 1 -- 172.21.15.2:0/1760564962 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8004d20 con 0x55b3afadfb20 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84b5600640 1 -- 172.21.15.2:0/1760564962 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84b8004e80 con 0x55b3afadfb20 2024-03-11T18:42:01.765 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.762+0000 7f84b5600640 1 -- 172.21.15.2:0/1760564962 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8005280 con 0x55b3afadfb20 2024-03-11T18:42:01.766 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 -- 172.21.15.2:0/1760564962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 msgr2=0x55b3afade870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009f50 tx=0x7f84b80118d0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 -- 172.21.15.2:0/1760564962 shutdown_connections 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afadf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1760564962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afade870 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.763+0000 7f84c9b76740 1 -- 172.21.15.2:0/1760564962 >> 172.21.15.2:0/1760564962 conn(0x55b3afaa64b0 msgr2=0x55b3afad9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84c9b76740 1 -- 172.21.15.2:0/1760564962 shutdown_connections 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84bcc00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 0x7f84a40a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84c9b76740 1 -- 172.21.15.2:0/1760564962 wait complete. 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84bcc00640 1 -- 172.21.15.2:0/4282696715 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f84ac014070 con 0x7f84c01086a0 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84bfe00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 0x7f84a40a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.764+0000 7f84c9b76740 1 -- start start 2024-03-11T18:42:01.767 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 Processor -- start 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84bfe00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 0x7f84a40a3f60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009900 tx=0x7f84b0005b50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afae8330 0x55b3afae5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afa455d0 con 0x55b3afae8330 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3af9eab50 con 0x55b3afadfb20 2024-03-11T18:42:01.769 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3af9ea7f0 con 0x55b3afadedb0 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84c9b76740 10 client.0 _ll_register_callbacks cb 0x55b3af9fe550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:43094/0 (socket says 172.21.15.2:43094) 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84bfe00640 1 -- 172.21.15.2:0/2198735940 learned_addr learned my addr 172.21.15.2:0/2198735940 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.765+0000 7f84bf400640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84c4e00640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afae8330 0x55b3afae5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 -- 172.21.15.2:0/2198735940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 msgr2=0x55b3afae5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 -- 172.21.15.2:0/2198735940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afae8330 msgr2=0x55b3afae5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afae8330 0x55b3afae5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 -- 172.21.15.2:0/2198735940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b8009c00 con 0x55b3afadfb20 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bfe00640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:01.770 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84bf400640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5950 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0cb350 tx=0x7f84ac09a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a15ece2228e5ae8b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.766+0000 7f84aa000640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:42:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.767+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac09e980 con 0x55b3afadfb20 2024-03-11T18:42:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.767+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f84ac0a2fd0 con 0x55b3afadfb20 2024-03-11T18:42:01.771 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.767+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3afa17cd0 con 0x55b3afadfb20 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84ac0046c0 con 0x55b3afadfb20 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 10 client.26150 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b3afa43c50 con 0x55b3afadfb20 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b3afa17cd0 con 0x55b3afadfb20 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f84ac01f060 con 0x55b3afadfb20 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 10 client.26150 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 10 client.26150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 20 client.26150 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.768+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afaef2b0 0x55b3afaf16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.769+0000 7f84c9b76740 4 client.26150 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.769+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3afa3bce0 con 0x55b3afaef2b0 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.769+0000 7f84bfe00640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afaef2b0 0x55b3afaf16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.773 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.769+0000 7f84bfe00640 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afaef2b0 0x55b3afaf16d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca1842a2551f889d server_cookie=af27469d3a7f813c in_seq=0 out_seq=0 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.769+0000 7f84aa000640 10 client.26150 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.770+0000 7f84aa000640 1 -- 172.21.15.2:0/2198735940 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b3afa3bce0 con 0x55b3afaef2b0 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.770+0000 7f84aa000640 10 client.26150 handle_command_reply: tid=0 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.770+0000 7f84c9b76740 1 client.26150 shutdown 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.770+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 msgr2=0x55b3afae5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.770+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5950 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f84ac0cb350 tx=0x7f84ac09a250 comp rx=0 tx=0).stop 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 shutdown_connections 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afaef2b0 0x55b3afaf16d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afae8330 0x55b3afae5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5950 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2198735940 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afae5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84aa000640 0 client.26150 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 >> 172.21.15.2:0/2198735940 conn(0x55b3afaa64b0 msgr2=0x55b3afadb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 shutdown_connections 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 1 -- 172.21.15.2:0/2198735940 wait complete. 2024-03-11T18:42:01.774 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.771+0000 7f84c9b76740 20 client.26150 trim_cache size 0 max 16384 2024-03-11T18:42:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 Processor -- start 2024-03-11T18:42:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 -- start start 2024-03-11T18:42:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afae7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.791 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3af78de60 con 0x55b3afadedb0 2024-03-11T18:42:01.792 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afae1020 con 0x55b3afadfb20 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.789+0000 7f84c9b76740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afade080 con 0x55b3afae8330 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afae7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84c4e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84c4e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:43110/0 (socket says 172.21.15.2:43110) 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84c4e00640 1 -- 172.21.15.2:0/2359404383 learned_addr learned my addr 172.21.15.2:0/2359404383 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 -- 172.21.15.2:0/2359404383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 msgr2=0x55b3afaf6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 -- 172.21.15.2:0/2359404383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 msgr2=0x55b3afae7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afae7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84bf400640 1 -- 172.21.15.2:0/2359404383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3af9e0530 con 0x55b3afadfb20 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.790+0000 7f84c4e00640 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:01.793 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.791+0000 7f84bf400640 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5cb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f84ac096c40 tx=0x7f84ac098d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=add6cb06a9fdc2e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.791+0000 7f84b4c00640 1 -- 172.21.15.2:0/2359404383 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0a2fd0 con 0x55b3afadfb20 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.791+0000 7f84b4c00640 1 -- 172.21.15.2:0/2359404383 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84ac09e980 con 0x55b3afadfb20 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.791+0000 7f84b4c00640 1 -- 172.21.15.2:0/2359404383 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0cb5d0 con 0x55b3afadfb20 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- 172.21.15.2:0/2359404383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 msgr2=0x55b3afae5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5cb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f84ac096c40 tx=0x7f84ac098d40 comp rx=0 tx=0).stop 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- 172.21.15.2:0/2359404383 shutdown_connections 2024-03-11T18:42:01.795 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afae8330 0x55b3afaf6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afadfb20 0x55b3afae5cb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 --2- 172.21.15.2:0/2359404383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadedb0 0x55b3afae7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- 172.21.15.2:0/2359404383 >> 172.21.15.2:0/2359404383 conn(0x55b3afaa64b0 msgr2=0x55b3afae3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- 172.21.15.2:0/2359404383 shutdown_connections 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- 172.21.15.2:0/2359404383 wait complete. 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.792+0000 7f84c9b76740 1 -- start start 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 Processor -- start 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afaf6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 0x55b3afae2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afae3460 con 0x55b3afadfb20 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afafb330 con 0x55b3afae8330 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3afafb4f0 con 0x55b3afadedb0 2024-03-11T18:42:01.796 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.793+0000 7f84c9b76740 10 client.0 _ll_register_callbacks cb 0x7f84ac00cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84c4e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 0x55b3afae2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afaf6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54834/0 (socket says 172.21.15.2:54834) 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bf400640 1 -- 172.21.15.2:0/1537279008 learned_addr learned my addr 172.21.15.2:0/1537279008 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 -- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 msgr2=0x55b3afae2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 0x55b3afae2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 -- 172.21.15.2:0/1537279008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 msgr2=0x55b3afae2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 -- 172.21.15.2:0/1537279008 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b3af9c6b00 con 0x55b3afadedb0 2024-03-11T18:42:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84c4e00640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 0x55b3afae2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bf400640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84bfe00640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afaf6550 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55b3afaf1f80 tx=0x7f84b0041070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80635c2774e2e782 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.794+0000 7f84aa000640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0050040 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9488+0+0 (secure 0 0 0) 0x7f84b0056580 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3afa6ad30 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b002dea0 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84c9b76740 10 client.26146 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b3afa43c50 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.795+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b3afa6ad30 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f84b0047040 con 0x55b3afadedb0 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 10 client.26146 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:01.800 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 20 client.26146 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afb60ad0 0x55b3afb62ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 4 client.26146 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b3af745600 con 0x55b3afb60ad0 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.796+0000 7f84bf400640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afb60ad0 0x55b3afb62ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.797+0000 7f84bf400640 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afb60ad0 0x55b3afb62ef0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4704dccf04671048 server_cookie=6b90da6a37a51738 in_seq=0 out_seq=0 2024-03-11T18:42:01.801 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:01.797+0000 7f84aa000640 10 client.26146 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:02.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:01 smithi002 ceph-mon[31328]: pgmap v258: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-03-11T18:42:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.010+0000 7f84aa000640 1 -- 172.21.15.2:0/1537279008 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b3af745600 con 0x55b3afb60ad0 2024-03-11T18:42:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.010+0000 7f84aa000640 10 client.26146 handle_command_reply: tid=0 2024-03-11T18:42:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.010+0000 7f84c9b76740 1 client.26146 shutdown 2024-03-11T18:42:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 msgr2=0x55b3afaf6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afaf6550 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55b3afaf1f80 tx=0x7f84b0041070 comp rx=0 tx=0).stop 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 shutdown_connections 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55b3afb60ad0 0x55b3afb62ef0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3afae8330 0x55b3afae2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55b3afadfb20 0x55b3afae2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 --2- 172.21.15.2:0/1537279008 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55b3afadedb0 0x55b3afaf6550 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84aa000640 0 client.26146 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 >> 172.21.15.2:0/1537279008 conn(0x55b3afaa64b0 msgr2=0x55b3afad84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 shutdown_connections 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.011+0000 7f84c9b76740 1 -- 172.21.15.2:0/1537279008 wait complete. 2024-03-11T18:42:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.012+0000 7f84c9b76740 20 client.26146 trim_cache size 0 max 16384 2024-03-11T18:42:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:42:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:42:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:42:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 msgr2=0x7f84a40a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 0x7f84a40a3f60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009900 tx=0x7f84b0005b50 comp rx=0 tx=0).stop 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 msgr2=0x7f84c019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00e9f0 tx=0x7f84ac00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 shutdown_connections 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f84a40a1aa0 0x7f84a40a3f60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84c01998b0 0x7f84c0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f84c01086a0 0x7f84c019fbf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 --2- 172.21.15.2:0/4282696715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84c01035d0 0x7f84c019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.013+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 >> 172.21.15.2:0/4282696715 conn(0x7f84c00fee70 msgr2=0x7f84c01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.014+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 shutdown_connections 2024-03-11T18:42:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:02.014+0000 7f84c6e00640 1 -- 172.21.15.2:0/4282696715 wait complete. 2024-03-11T18:42:02.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:01 smithi165 ceph-mon[36634]: pgmap v258: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-03-11T18:42:02.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:01 smithi023 ceph-mon[36406]: pgmap v258: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 222 op/s 2024-03-11T18:42:02.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 633200 7.67 MB/sec execute 302 sec latency 222.025 ms 2024-03-11T18:42:03.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 633725 7.66 MB/sec execute 303 sec latency 152.721 ms 2024-03-11T18:42:04.002 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:03 smithi002 ceph-mon[31328]: pgmap v259: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2024-03-11T18:42:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:03 smithi002 ceph-mon[31328]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T18:42:04.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:03 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:42:04.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:03 smithi165 ceph-mon[36634]: pgmap v259: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2024-03-11T18:42:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:03 smithi165 ceph-mon[36634]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T18:42:04.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:03 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:42:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:03 smithi023 ceph-mon[36406]: pgmap v259: 129 pgs: 129 active+clean; 99 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 198 op/s 2024-03-11T18:42:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:03 smithi023 ceph-mon[36406]: osdmap e103: 12 total, 12 up, 12 in 2024-03-11T18:42:04.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:03 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:42:04.434 INFO:tasks.workunit.client.0.smithi002.stdout: 1 634126 7.64 MB/sec execute 304 sec latency 94.273 ms 2024-03-11T18:42:05.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 634657 7.62 MB/sec execute 305 sec latency 65.770 ms 2024-03-11T18:42:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:05 smithi002 ceph-mon[31328]: pgmap v261: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:05 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:42:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:05 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:42:06.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:05 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:42:06.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:05 smithi165 ceph-mon[36634]: pgmap v261: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:05 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:05 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:05 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:05 smithi023 ceph-mon[36406]: pgmap v261: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:05 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:05 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:42:06.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:05 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:42:06.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 635491 7.62 MB/sec execute 306 sec latency 137.982 ms 2024-03-11T18:42:07.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 636179 7.61 MB/sec execute 307 sec latency 33.541 ms 2024-03-11T18:42:08.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:07 smithi002 ceph-mon[31328]: pgmap v262: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:08.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:07 smithi165 ceph-mon[36634]: pgmap v262: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:08.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:07 smithi023 ceph-mon[36406]: pgmap v262: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 90 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-03-11T18:42:08.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 637135 7.60 MB/sec execute 308 sec latency 34.602 ms 2024-03-11T18:42:09.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 638268 7.58 MB/sec execute 309 sec latency 26.783 ms 2024-03-11T18:42:10.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:09 smithi002 ceph-mon[31328]: pgmap v263: 129 pgs: 6 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 103 active+clean; 89 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 227 op/s 2024-03-11T18:42:10.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:09 smithi165 ceph-mon[36634]: pgmap v263: 129 pgs: 6 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 103 active+clean; 89 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 227 op/s 2024-03-11T18:42:10.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:09 smithi023 ceph-mon[36406]: pgmap v263: 129 pgs: 6 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 103 active+clean; 89 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 1.6 MiB/s wr, 227 op/s 2024-03-11T18:42:10.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 639650 7.59 MB/sec execute 310 sec latency 14.586 ms 2024-03-11T18:42:11.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 640515 7.58 MB/sec execute 311 sec latency 42.719 ms 2024-03-11T18:42:12.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:11 smithi165 ceph-mon[36634]: pgmap v264: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:12.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:11 smithi023 ceph-mon[36406]: pgmap v264: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:12.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:11 smithi002 ceph-mon[31328]: pgmap v264: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:12.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 641815 7.57 MB/sec execute 312 sec latency 60.565 ms 2024-03-11T18:42:13.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 643194 7.58 MB/sec execute 313 sec latency 90.648 ms 2024-03-11T18:42:14.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:13 smithi165 ceph-mon[36634]: pgmap v265: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:14.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:13 smithi165 ceph-mon[36634]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T18:42:14.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:13 smithi023 ceph-mon[36406]: pgmap v265: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:14.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:13 smithi023 ceph-mon[36406]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T18:42:14.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:13 smithi002 ceph-mon[31328]: pgmap v265: 129 pgs: 129 active+clean; 91 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-03-11T18:42:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:13 smithi002 ceph-mon[31328]: osdmap e104: 12 total, 12 up, 12 in 2024-03-11T18:42:14.436 INFO:tasks.workunit.client.0.smithi002.stdout: 1 644225 7.57 MB/sec execute 314 sec latency 33.044 ms 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 -- 172.21.15.2:0/1378164905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 msgr2=0x7fa1a8073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1378164905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a8073910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa190009900 tx=0x7fa19002bcd0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 -- 172.21.15.2:0/1378164905 shutdown_connections 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1378164905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a810d820 0x7fa1a810fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1378164905 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a8073e50 0x7fa1a810d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1378164905 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a8073910 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.180 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 -- 172.21.15.2:0/1378164905 >> 172.21.15.2:0/1378164905 conn(0x7fa1a80fca90 msgr2=0x7fa1a80feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 -- 172.21.15.2:0/1378164905 shutdown_connections 2024-03-11T18:42:15.181 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.176+0000 7fa1a7200640 1 -- 172.21.15.2:0/1378164905 wait complete. 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 Processor -- start 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 -- start start 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a8073e50 0x7fa1a819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a810d820 0x7fa1a8199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8199df0 con 0x7fa1a8073e50 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8199f60 con 0x7fa1a8073530 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a7200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a819a120 con 0x7fa1a810d820 2024-03-11T18:42:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50060/0 (socket says 172.21.15.2:50060) 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a5e00640 1 -- 172.21.15.2:0/1602736898 learned_addr learned my addr 172.21.15.2:0/1602736898 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a5400640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a8073e50 0x7fa1a819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.178+0000 7fa1a6800640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a810d820 0x7fa1a8199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 -- 172.21.15.2:0/1602736898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a810d820 msgr2=0x7fa1a8199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a810d820 0x7fa1a8199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 -- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a8073e50 msgr2=0x7fa1a819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a8073e50 0x7fa1a819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 -- 172.21.15.2:0/1602736898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190009560 con 0x7fa1a8073530 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa19002d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.179+0000 7fa19ea00640 1 -- 172.21.15.2:0/1602736898 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19002dd50 con 0x7fa1a8073530 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.180+0000 7fa19ea00640 1 -- 172.21.15.2:0/1602736898 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1900052d0 con 0x7fa1a8073530 2024-03-11T18:42:15.184 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.180+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a819a3a0 con 0x7fa1a8073530 2024-03-11T18:42:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.180+0000 7fa19ea00640 1 -- 172.21.15.2:0/1602736898 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa190003c90 con 0x7fa1a8073530 2024-03-11T18:42:15.185 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.180+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a81a6520 con 0x7fa1a8073530 2024-03-11T18:42:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 Processor -- start 2024-03-11T18:42:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa19ea00640 1 -- 172.21.15.2:0/1602736898 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fa190005440 con 0x7fa1a8073530 2024-03-11T18:42:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 -- start start 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef7d60 0x558087ef6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087f02960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087eb2620 con 0x558087f00570 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087e648c0 con 0x558087ef6ff0 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1b0033740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087e61df0 con 0x558087ef7d60 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1a6800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef7d60 0x558087ef6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50062/0 (socket says 172.21.15.2:50062) 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.184+0000 7fa1a5e00640 1 -- 172.21.15.2:0/2756821320 learned_addr learned my addr 172.21.15.2:0/2756821320 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5400640 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087f02960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 -- 172.21.15.2:0/2756821320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef7d60 msgr2=0x558087ef6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef7d60 0x558087ef6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 -- 172.21.15.2:0/2756821320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 msgr2=0x558087f02960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087f02960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 -- 172.21.15.2:0/2756821320 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558087ddeb00 con 0x558087ef6ff0 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5400640 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087f02960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:42:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa1900ed520 tx=0x7fa1900f3f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54347cef331f5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.185+0000 7fa197400640 1 -- 172.21.15.2:0/2756821320 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1900f8ae0 con 0x558087ef6ff0 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.186+0000 7fa197400640 1 -- 172.21.15.2:0/2756821320 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1900f3d80 con 0x558087ef6ff0 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.186+0000 7fa197400640 1 -- 172.21.15.2:0/2756821320 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1900fe700 con 0x558087ef6ff0 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.186+0000 7fa19ea00640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 0x7fa18c0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.186+0000 7fa19ea00640 1 -- 172.21.15.2:0/1602736898 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fa1900f0200 con 0x7fa1a8073530 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.186+0000 7fa1a6800640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 0x7fa18c0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 -- 172.21.15.2:0/2756821320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 msgr2=0x558087ef7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa1900ed520 tx=0x7fa1900f3f90 comp rx=0 tx=0).stop 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 -- 172.21.15.2:0/2756821320 shutdown_connections 2024-03-11T18:42:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087f02960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef6ff0 0x558087ef7470 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 --2- 172.21.15.2:0/2756821320 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef7d60 0x558087ef6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 -- 172.21.15.2:0/2756821320 >> 172.21.15.2:0/2756821320 conn(0x558087ebe4b0 msgr2=0x558087ef1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 -- 172.21.15.2:0/2756821320 shutdown_connections 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1a6800640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 0x7fa18c0a4040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0005fd0 tx=0x7fa1a00096a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.187+0000 7fa1b0033740 1 -- 172.21.15.2:0/2756821320 wait complete. 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.188+0000 7fa1b0033740 1 -- start start 2024-03-11T18:42:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.188+0000 7fa1b0033740 1 Processor -- start 2024-03-11T18:42:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087e5d5d0 con 0x558087f00570 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087e02b50 con 0x558087ef7d60 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087e027f0 con 0x558087ef6ff0 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1b0033740 10 client.0 _ll_register_callbacks cb 0x558087e16550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a5e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:48914/0 (socket says 172.21.15.2:48914) 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50076/0 (socket says 172.21.15.2:50076) 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.189+0000 7fa1a5e00640 1 -- 172.21.15.2:0/3099891675 learned_addr learned my addr 172.21.15.2:0/3099891675 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 -- 172.21.15.2:0/3099891675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 msgr2=0x558087efd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 -- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 msgr2=0x558087efdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 -- 172.21.15.2:0/3099891675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a81a6520 con 0x558087f00570 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a5400640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa1a6800640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efe080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0011db0 tx=0x7fa1a0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d52305a089b133 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa18be00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.190+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a0016070 con 0x558087f00570 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.191+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fa1a0003c40 con 0x558087f00570 2024-03-11T18:42:15.195 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.191+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558087e2fcd0 con 0x558087f00570 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.191+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1a0003e00 con 0x558087f00570 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.191+0000 7fa1b0033740 10 client.16014 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.191+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558087e5bc50 con 0x558087f00570 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.192+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558087e2fcd0 con 0x558087f00570 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa1a001f9b0 con 0x558087f00570 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 10 client.16014 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:15.196 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 20 client.16014 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f18c60 0x558087f1b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 4 client.16014 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.193+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558087e53ce0 con 0x558087f18c60 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.194+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f18c60 0x558087f1b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.194+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f18c60 0x558087f1b080 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89baccf2e6e1f010 server_cookie=25608a22b4983f1c in_seq=0 out_seq=0 2024-03-11T18:42:15.197 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.194+0000 7fa18be00640 10 client.16014 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.195+0000 7fa18be00640 1 -- 172.21.15.2:0/3099891675 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558087e53ce0 con 0x558087f18c60 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.195+0000 7fa18be00640 10 client.16014 handle_command_reply: tid=0 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.195+0000 7fa1b0033740 1 client.16014 shutdown 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 msgr2=0x558087efe080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efe080 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0011db0 tx=0x7fa1a0004ab0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 shutdown_connections 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f18c60 0x558087f1b080 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efe080 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3099891675 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087efd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 >> 172.21.15.2:0/3099891675 conn(0x558087ebe4b0 msgr2=0x558087ef3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa18be00640 0 client.16014 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 shutdown_connections 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 1 -- 172.21.15.2:0/3099891675 wait complete. 2024-03-11T18:42:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.196+0000 7fa1b0033740 20 client.16014 trim_cache size 0 max 16384 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.213+0000 7fa1b0033740 1 Processor -- start 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.213+0000 7fa1b0033740 1 -- start start 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087effd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087ba5e60 con 0x558087f00570 2024-03-11T18:42:15.218 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087ef2ce0 con 0x558087ef7d60 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1b0033740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087ef8490 con 0x558087ef6ff0 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50088/0 (socket says 172.21.15.2:50088) 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 -- 172.21.15.2:0/1173423053 learned_addr learned my addr 172.21.15.2:0/1173423053 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a6800640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087effd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 -- 172.21.15.2:0/1173423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 msgr2=0x558087effd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087effd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 -- 172.21.15.2:0/1173423053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 msgr2=0x558087efbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a5400640 1 -- 172.21.15.2:0/1173423053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558087df8530 con 0x558087ef7d60 2024-03-11T18:42:15.219 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.214+0000 7fa1a6800640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:42:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.215+0000 7fa1a5400640 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa198005fd0 tx=0x7fa19800bb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efadee9bc9649515 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.215+0000 7fa196a00640 1 -- 172.21.15.2:0/1173423053 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa198017070 con 0x558087ef7d60 2024-03-11T18:42:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.215+0000 7fa196a00640 1 -- 172.21.15.2:0/1173423053 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa198003de0 con 0x558087ef7d60 2024-03-11T18:42:15.220 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.215+0000 7fa196a00640 1 -- 172.21.15.2:0/1173423053 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa198011e40 con 0x558087ef7d60 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- 172.21.15.2:0/1173423053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 msgr2=0x558087efdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa198005fd0 tx=0x7fa19800bb00 comp rx=0 tx=0).stop 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- 172.21.15.2:0/1173423053 shutdown_connections 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087f00570 0x558087efbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087efdea0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 --2- 172.21.15.2:0/1173423053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087ef6ff0 0x558087effd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.221 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- 172.21.15.2:0/1173423053 >> 172.21.15.2:0/1173423053 conn(0x558087ebe4b0 msgr2=0x558087ef1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- 172.21.15.2:0/1173423053 shutdown_connections 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- 172.21.15.2:0/1173423053 wait complete. 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.216+0000 7fa1b0033740 1 -- start start 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 Processor -- start 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087ef6ff0 0x558087f0dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087f00570 0x558087f680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087f68630 con 0x558087ef6ff0 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087f68770 con 0x558087ef7d60 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x558087f68930 con 0x558087f00570 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.217+0000 7fa1b0033740 10 client.0 _ll_register_callbacks cb 0x7fa1a0009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:50090/0 (socket says 172.21.15.2:50090) 2024-03-11T18:42:15.222 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 -- 172.21.15.2:0/3123880138 learned_addr learned my addr 172.21.15.2:0/3123880138 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087ef6ff0 0x558087f0dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a6800640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087f00570 0x558087f680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 -- 172.21.15.2:0/3123880138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087f00570 msgr2=0x558087f680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087f00570 0x558087f680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 -- 172.21.15.2:0/3123880138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087ef6ff0 msgr2=0x558087f0dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087ef6ff0 0x558087f0dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.223 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.218+0000 7fa1a5400640 1 -- 172.21.15.2:0/3123880138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198004360 con 0x558087ef7d60 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa1a5400640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa198000c00 tx=0x7fa198003b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61081f53b4627469 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa18be00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa198028070 con 0x558087ef7d60 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558087e82d30 con 0x558087ef7d60 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fa198011650 con 0x558087ef7d60 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa198017070 con 0x558087ef7d60 2024-03-11T18:42:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa1b0033740 10 client.26174 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.219+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558087e1e2c0 con 0x558087ef7d60 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558087e82d30 con 0x558087ef7d60 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa19802b020 con 0x558087ef7d60 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 10 client.26174 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 10 client.26174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 20 client.26174 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f701a0 0x558087f725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 4 client.26174 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558087b5d600 con 0x558087f701a0 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.220+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f701a0 0x558087f725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.221+0000 7fa1a5e00640 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f701a0 0x558087f725c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8c7bb6aded3785 server_cookie=37437c2f820ef037 in_seq=0 out_seq=0 2024-03-11T18:42:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.221+0000 7fa18be00640 10 client.26174 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:15.435 INFO:tasks.workunit.client.0.smithi002.stdout: 1 645297 7.56 MB/sec execute 315 sec latency 104.701 ms 2024-03-11T18:42:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa18be00640 1 -- 172.21.15.2:0/3123880138 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558087b5d600 con 0x558087f701a0 2024-03-11T18:42:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa18be00640 10 client.26174 handle_command_reply: tid=0 2024-03-11T18:42:15.451 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 client.26174 shutdown 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 msgr2=0x558087f0e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa198000c00 tx=0x7fa198003b30 comp rx=0 tx=0).stop 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 shutdown_connections 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x558087f701a0 0x558087f725c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x558087f00570 0x558087f680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558087ef7d60 0x558087f0e1a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 --2- 172.21.15.2:0/3123880138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558087ef6ff0 0x558087f0dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 >> 172.21.15.2:0/3123880138 conn(0x558087ebe4b0 msgr2=0x558087efbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.448+0000 7fa18be00640 0 client.26174 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.449+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 shutdown_connections 2024-03-11T18:42:15.452 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.449+0000 7fa1b0033740 1 -- 172.21.15.2:0/3123880138 wait complete. 2024-03-11T18:42:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.449+0000 7fa1b0033740 20 client.26174 trim_cache size 0 max 16384 2024-03-11T18:42:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:42:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:42:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:42:15.453 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:42:15.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.452+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 msgr2=0x7fa18c0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.455 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.452+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 0x7fa18c0a4040 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0005fd0 tx=0x7fa1a00096a0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.452+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 msgr2=0x7fa1a819f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.452+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa190000c00 tx=0x7fa19002d890 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 shutdown_connections 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1a810d820 0x7fa1a8199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fa18c0a1b80 0x7fa18c0a4040 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1a8073e50 0x7fa1a819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 --2- 172.21.15.2:0/1602736898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1a8073530 0x7fa1a819f5a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 >> 172.21.15.2:0/1602736898 conn(0x7fa1a80fca90 msgr2=0x7fa1a80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 shutdown_connections 2024-03-11T18:42:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:15.453+0000 7fa1a7200640 1 -- 172.21.15.2:0/1602736898 wait complete. 2024-03-11T18:42:16.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:15 smithi165 ceph-mon[36634]: pgmap v267: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:16.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:15 smithi023 ceph-mon[36406]: pgmap v267: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:16.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:15 smithi002 ceph-mon[31328]: pgmap v267: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:16.436 INFO:tasks.workunit.client.0.smithi002.stdout: 1 646797 7.57 MB/sec execute 316 sec latency 25.512 ms 2024-03-11T18:42:17.436 INFO:tasks.workunit.client.0.smithi002.stdout: 1 647734 7.56 MB/sec execute 317 sec latency 74.646 ms 2024-03-11T18:42:18.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:17 smithi023 ceph-mon[36406]: pgmap v268: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:18.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:17 smithi165 ceph-mon[36634]: pgmap v268: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:18.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:17 smithi002 ceph-mon[31328]: pgmap v268: 129 pgs: 129 active+clean; 97 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 150 op/s 2024-03-11T18:42:18.436 INFO:tasks.workunit.client.0.smithi002.stdout: 1 648958 7.55 MB/sec execute 318 sec latency 36.301 ms 2024-03-11T18:42:19.436 INFO:tasks.workunit.client.0.smithi002.stdout: 1 650333 7.56 MB/sec execute 319 sec latency 17.737 ms 2024-03-11T18:42:20.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:19 smithi165 ceph-mon[36634]: pgmap v269: 129 pgs: 129 active+clean; 89 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-11T18:42:20.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:19 smithi023 ceph-mon[36406]: pgmap v269: 129 pgs: 129 active+clean; 89 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-11T18:42:20.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:19 smithi002 ceph-mon[31328]: pgmap v269: 129 pgs: 129 active+clean; 89 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-03-11T18:42:20.437 INFO:tasks.workunit.client.0.smithi002.stdout: 1 651384 7.55 MB/sec execute 320 sec latency 25.528 ms 2024-03-11T18:42:21.437 INFO:tasks.workunit.client.0.smithi002.stdout: 1 652913 7.55 MB/sec execute 321 sec latency 29.700 ms 2024-03-11T18:42:22.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:21 smithi165 ceph-mon[36634]: pgmap v270: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:22.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:21 smithi023 ceph-mon[36406]: pgmap v270: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:22.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:21 smithi002 ceph-mon[31328]: pgmap v270: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:22.437 INFO:tasks.workunit.client.0.smithi002.stdout: 1 653997 7.55 MB/sec execute 322 sec latency 46.137 ms 2024-03-11T18:42:23.437 INFO:tasks.workunit.client.0.smithi002.stdout: 1 655078 7.54 MB/sec execute 323 sec latency 33.527 ms 2024-03-11T18:42:24.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:23 smithi165 ceph-mon[36634]: pgmap v271: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:24.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:23 smithi023 ceph-mon[36406]: pgmap v271: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:24.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:23 smithi002 ceph-mon[31328]: pgmap v271: 129 pgs: 129 active+clean; 92 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-03-11T18:42:24.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 656764 7.55 MB/sec execute 324 sec latency 24.232 ms 2024-03-11T18:42:25.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 657809 7.54 MB/sec execute 325 sec latency 23.117 ms 2024-03-11T18:42:26.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:25 smithi165 ceph-mon[36634]: pgmap v272: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-11T18:42:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:25 smithi023 ceph-mon[36406]: pgmap v272: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-11T18:42:26.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:25 smithi002 ceph-mon[31328]: pgmap v272: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-03-11T18:42:26.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 659401 7.54 MB/sec execute 326 sec latency 24.538 ms 2024-03-11T18:42:27.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 660937 7.55 MB/sec execute 327 sec latency 122.662 ms 2024-03-11T18:42:28.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:27 smithi165 ceph-mon[36634]: pgmap v273: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.9 MiB/s wr, 222 op/s 2024-03-11T18:42:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:27 smithi023 ceph-mon[36406]: pgmap v273: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.9 MiB/s wr, 222 op/s 2024-03-11T18:42:28.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:27 smithi002 ceph-mon[31328]: pgmap v273: 129 pgs: 129 active+clean; 97 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 1.9 MiB/s wr, 222 op/s 2024-03-11T18:42:28.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 662023 7.54 MB/sec execute 328 sec latency 28.103 ms 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 -- 172.21.15.2:0/1105075701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1061d0 msgr2=0x7f982c069790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 --2- 172.21.15.2:0/1105075701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1061d0 0x7f982c069790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9820009900 tx=0x7f982002bc40 comp rx=0 tx=0).stop 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 -- 172.21.15.2:0/1105075701 shutdown_connections 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 --2- 172.21.15.2:0/1105075701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c103bc0 0x7f982c104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 --2- 172.21.15.2:0/1105075701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c069cd0 0x7f982c06a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 --2- 172.21.15.2:0/1105075701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1061d0 0x7f982c069790 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 -- 172.21.15.2:0/1105075701 >> 172.21.15.2:0/1105075701 conn(0x7f982c076e40 msgr2=0x7f982c077250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 -- 172.21.15.2:0/1105075701 shutdown_connections 2024-03-11T18:42:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.611+0000 7f9833c00640 1 -- 172.21.15.2:0/1105075701 wait complete. 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 Processor -- start 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 -- start start 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c103bc0 0x7f982c19e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 0x7f982c1a2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1a2c30 con 0x7f982c1061d0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1a2da0 con 0x7f982c069cd0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.613+0000 7f9833c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1a2f60 con 0x7f982c103bc0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43876/0 (socket says 172.21.15.2:43876) 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 -- 172.21.15.2:0/651888647 learned_addr learned my addr 172.21.15.2:0/651888647 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 0x7f982c1a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f982be00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c103bc0 0x7f982c19e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 -- 172.21.15.2:0/651888647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c103bc0 msgr2=0x7f982c19e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c103bc0 0x7f982c19e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 -- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 msgr2=0x7f982c1a2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 0x7f982c1a2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 -- 172.21.15.2:0/651888647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820009560 con 0x7f982c069cd0 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 0x7f982c1a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:42:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.614+0000 7f9831200640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f982002c250 tx=0x7f982002c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.615+0000 7f9829600640 1 -- 172.21.15.2:0/651888647 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820004410 con 0x7f982c069cd0 2024-03-11T18:42:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.615+0000 7f9829600640 1 -- 172.21.15.2:0/651888647 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98200052d0 con 0x7f982c069cd0 2024-03-11T18:42:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.615+0000 7f9829600640 1 -- 172.21.15.2:0/651888647 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820003c60 con 0x7f982c069cd0 2024-03-11T18:42:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.615+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c1a3240 con 0x7f982c069cd0 2024-03-11T18:42:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.615+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1aaac0 con 0x7f982c069cd0 2024-03-11T18:42:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.618+0000 7f9829600640 1 -- 172.21.15.2:0/651888647 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f9820005440 con 0x7f982c069cd0 2024-03-11T18:42:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.618+0000 7f98367da740 1 Processor -- start 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.618+0000 7f98367da740 1 -- start start 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb17b620 con 0x55d0fb1c0b20 2024-03-11T18:42:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb12d8c0 con 0x55d0fb1c9330 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f98367da740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb12adf0 con 0x55d0fb1bfdb0 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f9831c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f982be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f9831200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f982be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43880/0 (socket says 172.21.15.2:43880) 2024-03-11T18:42:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.619+0000 7f982be00640 1 -- 172.21.15.2:0/2834480424 learned_addr learned my addr 172.21.15.2:0/2834480424 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 -- 172.21.15.2:0/2834480424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 msgr2=0x55d0fb1c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 -- 172.21.15.2:0/2834480424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 msgr2=0x55d0fb1bf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1bf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 -- 172.21.15.2:0/2834480424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fb0a7b00 con 0x55d0fb1c9330 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f9831200640 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f9829600640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 0x7f98140a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f982be00640 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ca0 tx=0x7f9818011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=937bd635006b6d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f9829600640 1 -- 172.21.15.2:0/651888647 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f98200f03d0 con 0x7f982c069cd0 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f9831c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 0x7f98140a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f981e000640 1 -- 172.21.15.2:0/2834480424 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818019630 con 0x55d0fb1c9330 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f981e000640 1 -- 172.21.15.2:0/2834480424 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9818019c70 con 0x55d0fb1c9330 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.620+0000 7f981e000640 1 -- 172.21.15.2:0/2834480424 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981800ebc0 con 0x55d0fb1c9330 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 -- 172.21.15.2:0/2834480424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 msgr2=0x55d0fb1cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f9831c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 0x7f98140a40d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9824005fd0 tx=0x7f98240098a0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ca0 tx=0x7f9818011810 comp rx=0 tx=0).stop 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 -- 172.21.15.2:0/2834480424 shutdown_connections 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c9330 0x55d0fb1cb720 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 --2- 172.21.15.2:0/2834480424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1bf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 -- 172.21.15.2:0/2834480424 >> 172.21.15.2:0/2834480424 conn(0x55d0fb1874b0 msgr2=0x55d0fb1baaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 -- 172.21.15.2:0/2834480424 shutdown_connections 2024-03-11T18:42:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.621+0000 7f98367da740 1 -- 172.21.15.2:0/2834480424 wait complete. 2024-03-11T18:42:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.622+0000 7f98367da740 1 -- start start 2024-03-11T18:42:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.622+0000 7f98367da740 1 Processor -- start 2024-03-11T18:42:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb1265d0 con 0x55d0fb1bfdb0 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb0cbb50 con 0x55d0fb1c0b20 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb0cb7f0 con 0x55d0fb1c9330 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f98367da740 10 client.0 _ll_register_callbacks cb 0x55d0fb0df550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f982be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:49070/0 (socket says 172.21.15.2:49070) 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 -- 172.21.15.2:0/3790777711 learned_addr learned my addr 172.21.15.2:0/3790777711 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f982be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43882/0 (socket says 172.21.15.2:43882) 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 -- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 msgr2=0x55d0fb1c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 -- 172.21.15.2:0/3790777711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1bfdb0 msgr2=0x55d0fb1c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.623+0000 7f9831c00640 1 -- 172.21.15.2:0/3790777711 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98180098d0 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f9831c00640 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9824010a50 tx=0x7f9824004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5b644a6cbd414db server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f9812a00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982401ed40 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0fb0f8cd0 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f982401f420 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f982401eea0 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f98367da740 10 client.26176 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.624+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d0fb124c50 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d0fb0f8cd0 con 0x55d0fb1c9330 2024-03-11T18:42:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f982401c6e0 con 0x55d0fb1c9330 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 10 client.26176 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 10 client.26176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 20 client.26176 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb1d9460 0x55d0fb1db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 4 client.26176 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0fb11cce0 con 0x55d0fb1d9460 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.625+0000 7f9831200640 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb1d9460 0x55d0fb1db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.626+0000 7f9831200640 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb1d9460 0x55d0fb1db880 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d6c4a29eb414df9 server_cookie=d01245567ec6b46c in_seq=0 out_seq=0 2024-03-11T18:42:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.626+0000 7f9812a00640 10 client.26176 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.627+0000 7f9812a00640 1 -- 172.21.15.2:0/3790777711 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d0fb11cce0 con 0x55d0fb1d9460 2024-03-11T18:42:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.627+0000 7f9812a00640 10 client.26176 handle_command_reply: tid=0 2024-03-11T18:42:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.627+0000 7f98367da740 1 client.26176 shutdown 2024-03-11T18:42:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.627+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 msgr2=0x55d0fb1c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.627+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9824010a50 tx=0x7f9824004a40 comp rx=0 tx=0).stop 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 shutdown_connections 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb1d9460 0x55d0fb1db880 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb1c6df0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 --2- 172.21.15.2:0/3790777711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 >> 172.21.15.2:0/3790777711 conn(0x55d0fb1874b0 msgr2=0x55d0fb1bc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f9812a00640 0 client.26176 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 shutdown_connections 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 1 -- 172.21.15.2:0/3790777711 wait complete. 2024-03-11T18:42:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.628+0000 7f98367da740 20 client.26176 trim_cache size 0 max 16384 2024-03-11T18:42:28.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 Processor -- start 2024-03-11T18:42:28.646 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 -- start start 2024-03-11T18:42:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.647 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 0x55d0fb1d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fae6ee60 con 0x55d0fb1c9330 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb1c1fa0 con 0x55d0fb1bfdb0 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.644+0000 7f98367da740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb1d5e50 con 0x55d0fb1c0b20 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 0x55d0fb1d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43894/0 (socket says 172.21.15.2:43894) 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:49076/0 (socket says 172.21.15.2:49076) 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831200640 1 -- 172.21.15.2:0/3532745840 learned_addr learned my addr 172.21.15.2:0/3532745840 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 -- 172.21.15.2:0/3532745840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 msgr2=0x55d0fb1c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 -- 172.21.15.2:0/3532745840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 msgr2=0x55d0fb1d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 0x55d0fb1d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.648 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f982be00640 1 -- 172.21.15.2:0/3532745840 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0fb0c1530 con 0x55d0fb1c0b20 2024-03-11T18:42:28.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831200640 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:28.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.645+0000 7f9831c00640 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 0x55d0fb1d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:28.649 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.646+0000 7f982be00640 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9818011d10 tx=0x7f9818005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa32ea6940c02c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.646+0000 7f981d600640 1 -- 172.21.15.2:0/3532745840 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981800e5b0 con 0x55d0fb1c0b20 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.646+0000 7f981d600640 1 -- 172.21.15.2:0/3532745840 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f981800ed90 con 0x55d0fb1c0b20 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.646+0000 7f981d600640 1 -- 172.21.15.2:0/3532745840 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818003850 con 0x55d0fb1c0b20 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- 172.21.15.2:0/3532745840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 msgr2=0x55d0fb1c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9818011d10 tx=0x7f9818005db0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- 172.21.15.2:0/3532745840 shutdown_connections 2024-03-11T18:42:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c9330 0x55d0fb1d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1c6be0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 --2- 172.21.15.2:0/3532745840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- 172.21.15.2:0/3532745840 >> 172.21.15.2:0/3532745840 conn(0x55d0fb1874b0 msgr2=0x55d0fb1c4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- 172.21.15.2:0/3532745840 shutdown_connections 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- 172.21.15.2:0/3532745840 wait complete. 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 -- start start 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.647+0000 7f98367da740 1 Processor -- start 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb22f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb1cddb0 con 0x55d0fb1c0b20 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb22fec0 con 0x55d0fb1bfdb0 2024-03-11T18:42:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0fb230000 con 0x55d0fb1c9330 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f98367da740 10 client.0 _ll_register_callbacks cb 0x7f9824009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:43896/0 (socket says 172.21.15.2:43896) 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831200640 1 -- 172.21.15.2:0/2329995206 learned_addr learned my addr 172.21.15.2:0/2329995206 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb22f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f982be00640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 -- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 msgr2=0x55d0fb1cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 -- 172.21.15.2:0/2329995206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 msgr2=0x55d0fb1cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831c00640 1 -- 172.21.15.2:0/2329995206 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1a3240 con 0x55d0fb1c9330 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.648+0000 7f9831200640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f9831c00640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb22f980 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f982401cac0 tx=0x7f9824010680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61100f00465301ce server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f9812a00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:42:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982401ed40 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f9824004ab0 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0fb14bd30 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f982401eea0 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f98367da740 10 client.26182 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.649+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d0fb124c50 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d0fb14bd30 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f98240053e0 con 0x55d0fb1c9330 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 10 client.26182 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 10 client.26182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 20 client.26182 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb2317e0 0x55d0fb233c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 4 client.26182 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:42:28.658 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d0fae26600 con 0x55d0fb2317e0 2024-03-11T18:42:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.650+0000 7f9831200640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb2317e0 0x55d0fb233c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.651+0000 7f9831200640 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb2317e0 0x55d0fb233c00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6495edf7cc53766 server_cookie=67b70baaa5e3516a in_seq=0 out_seq=0 2024-03-11T18:42:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.651+0000 7f9812a00640 10 client.26182 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.895+0000 7f9812a00640 1 -- 172.21.15.2:0/2329995206 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d0fae26600 con 0x55d0fb2317e0 2024-03-11T18:42:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.895+0000 7f9812a00640 10 client.26182 handle_command_reply: tid=0 2024-03-11T18:42:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.895+0000 7f98367da740 1 client.26182 shutdown 2024-03-11T18:42:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.895+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 msgr2=0x55d0fb22f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.895+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb22f980 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f982401cac0 tx=0x7f9824010680 comp rx=0 tx=0).stop 2024-03-11T18:42:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 shutdown_connections 2024-03-11T18:42:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d0fb2317e0 0x55d0fb233c00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d0fb1c9330 0x55d0fb22f980 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d0fb1c0b20 0x55d0fb1cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 --2- 172.21.15.2:0/2329995206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d0fb1bfdb0 0x55d0fb1cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 >> 172.21.15.2:0/2329995206 conn(0x55d0fb1874b0 msgr2=0x55d0fb1b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f9812a00640 0 client.26182 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.896+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 shutdown_connections 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.897+0000 7f98367da740 1 -- 172.21.15.2:0/2329995206 wait complete. 2024-03-11T18:42:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.897+0000 7f98367da740 20 client.26182 trim_cache size 0 max 16384 2024-03-11T18:42:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:42:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:42:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:42:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:42:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 msgr2=0x7f98140a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 0x7f98140a40d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9824005fd0 tx=0x7f98240098a0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 msgr2=0x7f982c19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f982002c250 tx=0x7f982002c6d0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 shutdown_connections 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f98140a1c10 0x7f98140a40d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f982c1061d0 0x7f982c1a2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f982c103bc0 0x7f982c19e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 --2- 172.21.15.2:0/651888647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c069cd0 0x7f982c19ddc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 >> 172.21.15.2:0/651888647 conn(0x7f982c076e40 msgr2=0x7f982c1029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 shutdown_connections 2024-03-11T18:42:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:28.898+0000 7f9833c00640 1 -- 172.21.15.2:0/651888647 wait complete. 2024-03-11T18:42:29.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 663741 7.54 MB/sec execute 329 sec latency 118.809 ms 2024-03-11T18:42:30.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:29 smithi165 ceph-mon[36634]: pgmap v274: 129 pgs: 129 active+clean; 101 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 289 op/s 2024-03-11T18:42:30.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:29 smithi023 ceph-mon[36406]: pgmap v274: 129 pgs: 129 active+clean; 101 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 289 op/s 2024-03-11T18:42:30.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:29 smithi002 ceph-mon[31328]: pgmap v274: 129 pgs: 129 active+clean; 101 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 289 op/s 2024-03-11T18:42:30.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 664740 7.54 MB/sec execute 330 sec latency 40.833 ms 2024-03-11T18:42:31.438 INFO:tasks.workunit.client.0.smithi002.stdout: 1 665722 7.53 MB/sec execute 331 sec latency 23.605 ms 2024-03-11T18:42:32.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:31 smithi165 ceph-mon[36634]: pgmap v275: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 205 op/s 2024-03-11T18:42:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:31 smithi023 ceph-mon[36406]: pgmap v275: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 205 op/s 2024-03-11T18:42:32.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:31 smithi002 ceph-mon[31328]: pgmap v275: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 205 op/s 2024-03-11T18:42:32.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 667084 7.53 MB/sec execute 332 sec latency 74.650 ms 2024-03-11T18:42:33.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 668216 7.53 MB/sec execute 333 sec latency 22.211 ms 2024-03-11T18:42:34.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:33 smithi165 ceph-mon[36634]: pgmap v276: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-03-11T18:42:34.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:33 smithi023 ceph-mon[36406]: pgmap v276: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-03-11T18:42:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:33 smithi002 ceph-mon[31328]: pgmap v276: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-03-11T18:42:34.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 669281 7.52 MB/sec execute 334 sec latency 27.048 ms 2024-03-11T18:42:35.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 670915 7.53 MB/sec execute 335 sec latency 24.753 ms 2024-03-11T18:42:36.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:35 smithi165 ceph-mon[36634]: pgmap v277: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 296 op/s 2024-03-11T18:42:36.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:35 smithi023 ceph-mon[36406]: pgmap v277: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 296 op/s 2024-03-11T18:42:36.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:35 smithi002 ceph-mon[31328]: pgmap v277: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.7 MiB/s wr, 296 op/s 2024-03-11T18:42:36.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 671897 7.52 MB/sec execute 336 sec latency 23.665 ms 2024-03-11T18:42:37.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 672990 7.51 MB/sec execute 337 sec latency 37.950 ms 2024-03-11T18:42:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:37 smithi165 ceph-mon[36634]: pgmap v278: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 226 op/s 2024-03-11T18:42:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:37 smithi023 ceph-mon[36406]: pgmap v278: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 226 op/s 2024-03-11T18:42:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:37 smithi002 ceph-mon[31328]: pgmap v278: 129 pgs: 129 active+clean; 98 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 226 op/s 2024-03-11T18:42:38.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 674881 7.53 MB/sec execute 338 sec latency 40.758 ms 2024-03-11T18:42:39.439 INFO:tasks.workunit.client.0.smithi002.stdout: 1 675693 7.51 MB/sec execute 339 sec latency 28.849 ms 2024-03-11T18:42:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:39 smithi165 ceph-mon[36634]: pgmap v279: 129 pgs: 129 active+clean; 101 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 271 op/s 2024-03-11T18:42:40.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:39 smithi023 ceph-mon[36406]: pgmap v279: 129 pgs: 129 active+clean; 101 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 271 op/s 2024-03-11T18:42:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:39 smithi002 ceph-mon[31328]: pgmap v279: 129 pgs: 129 active+clean; 101 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 271 op/s 2024-03-11T18:42:40.440 INFO:tasks.workunit.client.0.smithi002.stdout: 1 677167 7.51 MB/sec execute 340 sec latency 23.756 ms 2024-03-11T18:42:41.440 INFO:tasks.workunit.client.0.smithi002.stdout: 1 678700 7.52 MB/sec execute 341 sec latency 14.808 ms 2024-03-11T18:42:42.065 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.063+0000 7f0847e00640 1 -- 172.21.15.2:0/1833530497 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c02c6b0 con 0x7f0848073520 2024-03-11T18:42:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.065+0000 7f0850400640 1 -- 172.21.15.2:0/1833530497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848073520 msgr2=0x7f0848073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.065+0000 7f0850400640 1 --2- 172.21.15.2:0/1833530497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848073520 0x7f0848073900 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f083c009900 tx=0x7f083c02bcf0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.066+0000 7f0850400640 1 -- 172.21.15.2:0/1833530497 shutdown_connections 2024-03-11T18:42:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.066+0000 7f0850400640 1 --2- 172.21.15.2:0/1833530497 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848112130 0x7f0848114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.066+0000 7f0850400640 1 --2- 172.21.15.2:0/1833530497 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f0848111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.070 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.066+0000 7f0850400640 1 --2- 172.21.15.2:0/1833530497 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848073520 0x7f0848073900 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.066+0000 7f0850400640 1 -- 172.21.15.2:0/1833530497 >> 172.21.15.2:0/1833530497 conn(0x7f08480fcbd0 msgr2=0x7f08480feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.068+0000 7f0850400640 1 -- 172.21.15.2:0/1833530497 shutdown_connections 2024-03-11T18:42:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.069+0000 7f0850400640 1 -- 172.21.15.2:0/1833530497 wait complete. 2024-03-11T18:42:42.073 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.071+0000 7f0850400640 1 Processor -- start 2024-03-11T18:42:42.076 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.074+0000 7f0850400640 1 -- start start 2024-03-11T18:42:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848073520 0x7f084819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f084819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08481a2d00 con 0x7f0848112130 2024-03-11T18:42:42.077 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08481a2e70 con 0x7f0848073e40 2024-03-11T18:42:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.075+0000 7f0850400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08481a3030 con 0x7f0848073520 2024-03-11T18:42:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.076+0000 7f084e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.076+0000 7f084e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55462/0 (socket says 172.21.15.2:55462) 2024-03-11T18:42:42.078 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.076+0000 7f084d000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f084819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.077+0000 7f084e400640 1 -- 172.21.15.2:0/3020376639 learned_addr learned my addr 172.21.15.2:0/3020376639 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:42.079 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.077+0000 7f084e400640 1 -- 172.21.15.2:0/3020376639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848073520 msgr2=0x7f084819de40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084da00640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848073520 0x7f084819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084e400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848073520 0x7f084819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084e400640 1 -- 172.21.15.2:0/3020376639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 msgr2=0x7f084819e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084e400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f084819e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084e400640 1 -- 172.21.15.2:0/3020376639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c009560 con 0x7f0848112130 2024-03-11T18:42:42.080 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.078+0000 7f084d000640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f084819e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:42:42.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.079+0000 7f084e400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f084000eb40 tx=0x7f084000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.079+0000 7f0846000640 1 -- 172.21.15.2:0/3020376639 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840004b20 con 0x7f0848112130 2024-03-11T18:42:42.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.080+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08481a3310 con 0x7f0848112130 2024-03-11T18:42:42.082 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.080+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08481aab90 con 0x7f0848112130 2024-03-11T18:42:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.081+0000 7f0846000640 1 -- 172.21.15.2:0/3020376639 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0840010910 con 0x7f0848112130 2024-03-11T18:42:42.083 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.081+0000 7f0846000640 1 -- 172.21.15.2:0/3020376639 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084000f8f0 con 0x7f0848112130 2024-03-11T18:42:42.089 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.088+0000 7f0846000640 1 -- 172.21.15.2:0/3020376639 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f0840010430 con 0x7f0848112130 2024-03-11T18:42:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.090+0000 7f0846000640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 0x7f08300a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.092 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.090+0000 7f0846000640 1 -- 172.21.15.2:0/3020376639 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f0840014070 con 0x7f0848112130 2024-03-11T18:42:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.094+0000 7f084da00640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 0x7f08300a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.095+0000 7f08531a6740 1 Processor -- start 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.095+0000 7f08531a6740 1 -- start start 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 0x55d4e44b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e4472620 con 0x55d4e44d1340 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e44248c0 con 0x55d4e44b7b20 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f08531a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e4421df0 con 0x55d4e44b6db0 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f084d000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f084d000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58932/0 (socket says 172.21.15.2:58932) 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.096+0000 7f084d000640 1 -- 172.21.15.2:0/1362593149 learned_addr learned my addr 172.21.15.2:0/1362593149 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.097+0000 7f084d000640 1 -- 172.21.15.2:0/1362593149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 msgr2=0x55d4e44b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.097+0000 7f084da00640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 0x55d4e44b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.097+0000 7f084d000640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 0x55d4e44b6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.097+0000 7f084d000640 1 -- 172.21.15.2:0/1362593149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 msgr2=0x55d4e44d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.100 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.097+0000 7f084e400640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.098+0000 7f084d000640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.098+0000 7f084d000640 1 -- 172.21.15.2:0/1362593149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4e439eb00 con 0x55d4e44b6db0 2024-03-11T18:42:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.100+0000 7f084da00640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 0x55d4e44b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:42:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.100+0000 7f084e400640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:42:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.100+0000 7f084d000640 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0834009c00 tx=0x7f0834010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4734fb72b22762f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.101+0000 7f083aa00640 1 -- 172.21.15.2:0/1362593149 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834004030 con 0x55d4e44b6db0 2024-03-11T18:42:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.101+0000 7f083aa00640 1 -- 172.21.15.2:0/1362593149 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08340035f0 con 0x55d4e44b6db0 2024-03-11T18:42:42.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.101+0000 7f083aa00640 1 -- 172.21.15.2:0/1362593149 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340126b0 con 0x55d4e44b6db0 2024-03-11T18:42:42.104 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.102+0000 7f08531a6740 1 -- 172.21.15.2:0/1362593149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 msgr2=0x55d4e44b7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.102+0000 7f08531a6740 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0834009c00 tx=0x7f0834010740 comp rx=0 tx=0).stop 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f084da00640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 0x7f08300a4050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f083c0058c0 tx=0x7f083c0046c0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 -- 172.21.15.2:0/1362593149 shutdown_connections 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44b7230 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 --2- 172.21.15.2:0/1362593149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b7b20 0x55d4e44b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 -- 172.21.15.2:0/1362593149 >> 172.21.15.2:0/1362593149 conn(0x55d4e447e4b0 msgr2=0x55d4e44b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 -- 172.21.15.2:0/1362593149 shutdown_connections 2024-03-11T18:42:42.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.103+0000 7f08531a6740 1 -- 172.21.15.2:0/1362593149 wait complete. 2024-03-11T18:42:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.105+0000 7f08531a6740 1 -- start start 2024-03-11T18:42:42.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.105+0000 7f08531a6740 1 Processor -- start 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.107+0000 7f08531a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e441d5d0 con 0x55d4e44d1340 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e43c2b50 con 0x55d4e44b6db0 2024-03-11T18:42:42.110 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e43c27f0 con 0x55d4e44b7b20 2024-03-11T18:42:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f08531a6740 10 client.0 _ll_register_callbacks cb 0x55d4e43d6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f084da00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.108+0000 7f084d000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084d000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58940/0 (socket says 172.21.15.2:58940) 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084d000640 1 -- 172.21.15.2:0/1846147186 learned_addr learned my addr 172.21.15.2:0/1846147186 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 -- 172.21.15.2:0/1846147186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 msgr2=0x55d4e44ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 -- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 msgr2=0x55d4e44ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.109+0000 7f084e400640 1 -- 172.21.15.2:0/1846147186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0834009830 con 0x55d4e44d1340 2024-03-11T18:42:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.111+0000 7f084e400640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44ced20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f08400cb6d0 tx=0x7f084009acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56556765a89a74a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.112+0000 7f082f400640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-03-11T18:42:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.112+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08400ce220 con 0x55d4e44d1340 2024-03-11T18:42:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.112+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f084009ae10 con 0x55d4e44d1340 2024-03-11T18:42:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.114+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4e43efcd0 con 0x55d4e44d1340 2024-03-11T18:42:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.114+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08400ce220 con 0x55d4e44d1340 2024-03-11T18:42:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.114+0000 7f08531a6740 10 client.16047 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.115+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d4e441bc50 con 0x55d4e44d1340 2024-03-11T18:42:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.115+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f084009afd0 con 0x55d4e44d1340 2024-03-11T18:42:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.117+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f084001f070 con 0x55d4e44d1340 2024-03-11T18:42:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 10 client.16047 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 20 client.16047 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e44f2380 0x55d4e44f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 4 client.16047 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:42:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.118+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4e43efcd0 con 0x55d4e44f2380 2024-03-11T18:42:42.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.119+0000 7f084da00640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e44f2380 0x55d4e44f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.121+0000 7f082f400640 10 client.16047 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:42.123 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.121+0000 7f084da00640 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e44f2380 0x55d4e44f47c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b4391dbf52d21ce server_cookie=d82de561f9a8e290 in_seq=0 out_seq=0 2024-03-11T18:42:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.122+0000 7f082f400640 1 -- 172.21.15.2:0/1846147186 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d4e43efcd0 con 0x55d4e44f2380 2024-03-11T18:42:42.124 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.122+0000 7f082f400640 10 client.16047 handle_command_reply: tid=0 2024-03-11T18:42:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.123+0000 7f08531a6740 1 client.16047 shutdown 2024-03-11T18:42:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.123+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 msgr2=0x55d4e44ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.125 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.123+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44ced20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f08400cb6d0 tx=0x7f084009acf0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 shutdown_connections 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e44f2380 0x55d4e44f47c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44ced20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 --2- 172.21.15.2:0/1846147186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f082f400640 0 client.16047 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:42.126 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.124+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 >> 172.21.15.2:0/1846147186 conn(0x55d4e447e4b0 msgr2=0x55d4e44b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.125+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 shutdown_connections 2024-03-11T18:42:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.126+0000 7f08531a6740 1 -- 172.21.15.2:0/1846147186 wait complete. 2024-03-11T18:42:42.128 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.126+0000 7f08531a6740 20 client.16047 trim_cache size 0 max 16384 2024-03-11T18:42:42.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:41 smithi165 ceph-mon[36634]: pgmap v280: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-11T18:42:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.154+0000 7f08531a6740 1 Processor -- start 2024-03-11T18:42:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:41 smithi023 ceph-mon[36406]: pgmap v280: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-11T18:42:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.154+0000 7f08531a6740 1 -- start start 2024-03-11T18:42:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.154+0000 7f08531a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.154+0000 7f08531a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f08531a6740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f08531a6740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e4165e60 con 0x55d4e44d1340 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f08531a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e44cf2d0 con 0x55d4e44b6db0 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f08531a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e44b2d00 con 0x55d4e44b7b20 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f084da00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f084da00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52390/0 (socket says 172.21.15.2:52390) 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f084da00640 1 -- 172.21.15.2:0/3786769596 learned_addr learned my addr 172.21.15.2:0/3786769596 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.155+0000 7f084d000640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.156+0000 7f084da00640 1 -- 172.21.15.2:0/3786769596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 msgr2=0x55d4e44c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.156+0000 7f084da00640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.156+0000 7f084da00640 1 -- 172.21.15.2:0/3786769596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 msgr2=0x55d4e44d8190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-11T18:42:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.157+0000 7f084e400640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.157+0000 7f084da00640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.157+0000 7f084da00640 1 -- 172.21.15.2:0/3786769596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4e43b8530 con 0x55d4e44b6db0 2024-03-11T18:42:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.157+0000 7f084d000640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.158+0000 7f084da00640 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f083c004fd0 tx=0x7f083c044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab63c44b9868110 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.158+0000 7f083a000640 1 -- 172.21.15.2:0/3786769596 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c005690 con 0x55d4e44b6db0 2024-03-11T18:42:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.158+0000 7f083a000640 1 -- 172.21.15.2:0/3786769596 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4e43efcd0 con 0x55d4e44b6db0 2024-03-11T18:42:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.158+0000 7f083a000640 1 -- 172.21.15.2:0/3786769596 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c034440 con 0x55d4e44b6db0 2024-03-11T18:42:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.159+0000 7f08531a6740 1 -- 172.21.15.2:0/3786769596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 msgr2=0x55d4e44d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.159+0000 7f08531a6740 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f083c004fd0 tx=0x7f083c044000 comp rx=0 tx=0).stop 2024-03-11T18:42:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.160+0000 7f08531a6740 1 -- 172.21.15.2:0/3786769596 shutdown_connections 2024-03-11T18:42:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.160+0000 7f08531a6740 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44d1340 0x55d4e44d8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.160+0000 7f08531a6740 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b7b20 0x55d4e44c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.160+0000 7f08531a6740 1 --2- 172.21.15.2:0/3786769596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44b6db0 0x55d4e44d09d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.161+0000 7f08531a6740 1 -- 172.21.15.2:0/3786769596 >> 172.21.15.2:0/3786769596 conn(0x55d4e447e4b0 msgr2=0x55d4e44f9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.161+0000 7f08531a6740 1 -- 172.21.15.2:0/3786769596 shutdown_connections 2024-03-11T18:42:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.161+0000 7f08531a6740 1 -- 172.21.15.2:0/3786769596 wait complete. 2024-03-11T18:42:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.162+0000 7f08531a6740 1 -- start start 2024-03-11T18:42:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.162+0000 7f08531a6740 1 Processor -- start 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 0x55d4e44d5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44d1340 0x55d4e45488a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e44d5c30 con 0x55d4e44b7b20 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e44d5da0 con 0x55d4e44d1340 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4e4548de0 con 0x55d4e44b6db0 2024-03-11T18:42:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f08531a6740 10 client.0 _ll_register_callbacks cb 0x7f084009b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f084da00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f084da00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58962/0 (socket says 172.21.15.2:58962) 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.163+0000 7f084da00640 1 -- 172.21.15.2:0/804659013 learned_addr learned my addr 172.21.15.2:0/804659013 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084d000640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 0x55d4e44d5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 -- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44d1340 msgr2=0x55d4e45488a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44d1340 0x55d4e45488a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 -- 172.21.15.2:0/804659013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 msgr2=0x55d4e44d5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 0x55d4e44d5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 -- 172.21.15.2:0/804659013 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4e439eb00 con 0x55d4e44b6db0 2024-03-11T18:42:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084d000640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 0x55d4e44d5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f084da00640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f083c005e10 tx=0x7f083c005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ec1e924fe5db1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.164+0000 7f082f400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c05c040 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f083c02cb60 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4e4442d30 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f083c005690 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f08531a6740 10 client.26197 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.165+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f083c038050 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4e4442d30 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f083c056020 con 0x55d4e44b6db0 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 10 client.26197 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 10 client.26197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 20 client.26197 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:42.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e454a1a0 0x55d4e454c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:42.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 4 client.26197 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:42:42.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d4e4413ce0 con 0x55d4e454a1a0 2024-03-11T18:42:42.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.166+0000 7f084d000640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e454a1a0 0x55d4e454c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:42.169 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.167+0000 7f084d000640 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e454a1a0 0x55d4e454c590 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e06b5a49d0cd5a server_cookie=e2e78abe12f35eb in_seq=0 out_seq=0 2024-03-11T18:42:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.167+0000 7f082f400640 10 client.26197 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:42.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:41 smithi002 ceph-mon[31328]: pgmap v280: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-03-11T18:42:42.440 INFO:tasks.workunit.client.0.smithi002.stdout: 1 679302 7.50 MB/sec execute 342 sec latency 204.492 ms 2024-03-11T18:42:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.480+0000 7f082f400640 1 -- 172.21.15.2:0/804659013 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d4e4413ce0 con 0x55d4e454a1a0 2024-03-11T18:42:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.481+0000 7f082f400640 10 client.26197 handle_command_reply: tid=0 2024-03-11T18:42:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.481+0000 7f08531a6740 1 client.26197 shutdown 2024-03-11T18:42:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.481+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 msgr2=0x55d4e44d4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.481+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f083c005e10 tx=0x7f083c005e40 comp rx=0 tx=0).stop 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 shutdown_connections 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55d4e454a1a0 0x55d4e454c590 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f082f400640 0 client.26197 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d4e44d1340 0x55d4e45488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55d4e44b7b20 0x55d4e44d5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 --2- 172.21.15.2:0/804659013 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55d4e44b6db0 0x55d4e44d4f20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 >> 172.21.15.2:0/804659013 conn(0x55d4e447e4b0 msgr2=0x55d4e44c95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.482+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 shutdown_connections 2024-03-11T18:42:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.483+0000 7f08531a6740 1 -- 172.21.15.2:0/804659013 wait complete. 2024-03-11T18:42:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.483+0000 7f08531a6740 20 client.26197 trim_cache size 0 max 16384 2024-03-11T18:42:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:42:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:42:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:42:42.487 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 msgr2=0x7f08300a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 0x7f08300a4050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f083c0058c0 tx=0x7f083c0046c0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 msgr2=0x7f08481a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f084000eb40 tx=0x7f084000c2d0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 shutdown_connections 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f08300a1b90 0x7f08300a4050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0848112130 0x7f08481a2700 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.488 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0848073e40 0x7f084819e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 --2- 172.21.15.2:0/3020376639 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0848073520 0x7f084819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 >> 172.21.15.2:0/3020376639 conn(0x7f08480fcbd0 msgr2=0x7f08480fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 shutdown_connections 2024-03-11T18:42:42.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:42.486+0000 7f0850400640 1 -- 172.21.15.2:0/3020376639 wait complete. 2024-03-11T18:42:43.440 INFO:tasks.workunit.client.0.smithi002.stdout: 1 680778 7.50 MB/sec execute 343 sec latency 55.080 ms 2024-03-11T18:42:44.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:43 smithi165 ceph-mon[36634]: pgmap v281: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 201 op/s 2024-03-11T18:42:44.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:43 smithi023 ceph-mon[36406]: pgmap v281: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 201 op/s 2024-03-11T18:42:44.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:43 smithi002 ceph-mon[31328]: pgmap v281: 129 pgs: 129 active+clean; 102 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 201 op/s 2024-03-11T18:42:44.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 682304 7.51 MB/sec execute 344 sec latency 21.946 ms 2024-03-11T18:42:45.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 683376 7.50 MB/sec execute 345 sec latency 25.872 ms 2024-03-11T18:42:46.155 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:45 smithi165 ceph-mon[36634]: pgmap v282: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-11T18:42:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:45 smithi023 ceph-mon[36406]: pgmap v282: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-11T18:42:46.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:45 smithi002 ceph-mon[31328]: pgmap v282: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-11T18:42:46.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 684880 7.50 MB/sec execute 346 sec latency 26.205 ms 2024-03-11T18:42:47.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 686019 7.50 MB/sec execute 347 sec latency 55.345 ms 2024-03-11T18:42:48.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:47 smithi165 ceph-mon[36634]: pgmap v283: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-03-11T18:42:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:47 smithi023 ceph-mon[36406]: pgmap v283: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-03-11T18:42:48.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:47 smithi002 ceph-mon[31328]: pgmap v283: 129 pgs: 129 active+clean; 94 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 1.5 MiB/s wr, 230 op/s 2024-03-11T18:42:48.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 687009 7.49 MB/sec execute 348 sec latency 39.287 ms 2024-03-11T18:42:49.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 688609 7.50 MB/sec execute 349 sec latency 24.491 ms 2024-03-11T18:42:50.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:49 smithi165 ceph-mon[36634]: pgmap v284: 129 pgs: 129 active+clean; 96 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 290 op/s 2024-03-11T18:42:50.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:49 smithi023 ceph-mon[36406]: pgmap v284: 129 pgs: 129 active+clean; 96 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 290 op/s 2024-03-11T18:42:50.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:49 smithi002 ceph-mon[31328]: pgmap v284: 129 pgs: 129 active+clean; 96 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 290 op/s 2024-03-11T18:42:50.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 689637 7.49 MB/sec execute 350 sec latency 24.842 ms 2024-03-11T18:42:51.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 690652 7.48 MB/sec execute 351 sec latency 28.096 ms 2024-03-11T18:42:52.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:51 smithi002 ceph-mon[31328]: pgmap v285: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.0 MiB/s wr, 266 op/s 2024-03-11T18:42:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:51 smithi165 ceph-mon[36634]: pgmap v285: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.0 MiB/s wr, 266 op/s 2024-03-11T18:42:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:51 smithi023 ceph-mon[36406]: pgmap v285: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.0 MiB/s wr, 266 op/s 2024-03-11T18:42:52.441 INFO:tasks.workunit.client.0.smithi002.stdout: 1 692298 7.49 MB/sec execute 352 sec latency 97.342 ms 2024-03-11T18:42:53.442 INFO:tasks.workunit.client.0.smithi002.stdout: 1 693202 7.48 MB/sec execute 353 sec latency 99.089 ms 2024-03-11T18:42:54.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:53 smithi002 ceph-mon[31328]: pgmap v286: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-11T18:42:54.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:53 smithi165 ceph-mon[36634]: pgmap v286: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-11T18:42:54.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:53 smithi023 ceph-mon[36406]: pgmap v286: 129 pgs: 129 active+clean; 98 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 1.7 MiB/s wr, 250 op/s 2024-03-11T18:42:54.442 INFO:tasks.workunit.client.0.smithi002.stdout: 1 694337 7.48 MB/sec execute 354 sec latency 25.603 ms 2024-03-11T18:42:55.442 INFO:tasks.workunit.client.0.smithi002.stdout: 1 696187 7.49 MB/sec execute 355 sec latency 20.391 ms 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 -- 172.21.15.2:0/3079719147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8541013a0 msgr2=0x7fe854101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 --2- 172.21.15.2:0/3079719147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8541013a0 0x7fe854101780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009ca0 tx=0x7fe84c02c100 comp rx=0 tx=0).stop 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 -- 172.21.15.2:0/3079719147 shutdown_connections 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 --2- 172.21.15.2:0/3079719147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe854106470 0x7fe85410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 --2- 172.21.15.2:0/3079719147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854101d50 0x7fe854105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 --2- 172.21.15.2:0/3079719147 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe8541013a0 0x7fe854101780 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.664+0000 7fe859200640 1 -- 172.21.15.2:0/3079719147 >> 172.21.15.2:0/3079719147 conn(0x7fe8540fcc00 msgr2=0x7fe8540ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.665+0000 7fe859200640 1 -- 172.21.15.2:0/3079719147 shutdown_connections 2024-03-11T18:42:55.668 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.665+0000 7fe859200640 1 -- 172.21.15.2:0/3079719147 wait complete. 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.665+0000 7fe859200640 1 Processor -- start 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 -- start start 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8541013a0 0x7fe85419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe854199880 0x7fe854199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85419a270 con 0x7fe8541013a0 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85419a3e0 con 0x7fe854106470 2024-03-11T18:42:55.670 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe859200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85419a5a0 con 0x7fe854199880 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe851400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe851400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37228/0 (socket says 172.21.15.2:37228) 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe852800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe854199880 0x7fe854199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe851400640 1 -- 172.21.15.2:0/1317159827 learned_addr learned my addr 172.21.15.2:0/1317159827 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.666+0000 7fe851e00640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8541013a0 0x7fe85419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 -- 172.21.15.2:0/1317159827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe854199880 msgr2=0x7fe854199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe854199880 0x7fe854199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 -- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8541013a0 msgr2=0x7fe85419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8541013a0 0x7fe85419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 -- 172.21.15.2:0/1317159827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c009900 con 0x7fe854106470 2024-03-11T18:42:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe851400640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9f0 tx=0x7fe83c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe84aa00640 1 -- 172.21.15.2:0/1317159827 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0096c0 con 0x7fe854106470 2024-03-11T18:42:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe84aa00640 1 -- 172.21.15.2:0/1317159827 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe83c005500 con 0x7fe854106470 2024-03-11T18:42:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8541a6590 con 0x7fe854106470 2024-03-11T18:42:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.667+0000 7fe84aa00640 1 -- 172.21.15.2:0/1317159827 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010430 con 0x7fe854106470 2024-03-11T18:42:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.668+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8541a6a70 con 0x7fe854106470 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.671+0000 7fe85be0d740 1 Processor -- start 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 -- start start 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe84aa00640 1 -- 172.21.15.2:0/1317159827 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fe83c005020 con 0x7fe854106470 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa09454620 con 0x55aa09499b20 2024-03-11T18:42:55.675 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa094068c0 con 0x55aa094a2330 2024-03-11T18:42:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe85be0d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa09403df0 con 0x55aa09498db0 2024-03-11T18:42:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe852800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe851e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe851400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe851e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:50812/0 (socket says 172.21.15.2:50812) 2024-03-11T18:42:55.678 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe852800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38652/0 (socket says 172.21.15.2:38652) 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.672+0000 7fe851e00640 1 -- 172.21.15.2:0/4123777455 learned_addr learned my addr 172.21.15.2:0/4123777455 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 -- 172.21.15.2:0/4123777455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 msgr2=0x55aa094a4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 -- 172.21.15.2:0/4123777455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 msgr2=0x55aa09498870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 -- 172.21.15.2:0/4123777455 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa09380b00 con 0x55aa09498db0 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe852800640 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe851e00640 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe84c037040 tx=0x7fe84c032d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88402e7a3a583491 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.673+0000 7fe843400640 1 -- 172.21.15.2:0/4123777455 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0326c0 con 0x55aa09498db0 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe843400640 1 -- 172.21.15.2:0/4123777455 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe84c032820 con 0x55aa09498db0 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe843400640 1 -- 172.21.15.2:0/4123777455 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c003e00 con 0x55aa09498db0 2024-03-11T18:42:55.679 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe84aa00640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 0x7fe8380a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe84aa00640 1 -- 172.21.15.2:0/1317159827 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fe83c014070 con 0x7fe854106470 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe852800640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 0x7fe8380a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 -- 172.21.15.2:0/4123777455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 msgr2=0x55aa09499230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe84c037040 tx=0x7fe84c032d80 comp rx=0 tx=0).stop 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 -- 172.21.15.2:0/4123777455 shutdown_connections 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa09499230 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 --2- 172.21.15.2:0/4123777455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa09498870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.674+0000 7fe85be0d740 1 -- 172.21.15.2:0/4123777455 >> 172.21.15.2:0/4123777455 conn(0x55aa094604b0 msgr2=0x55aa09493aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.675+0000 7fe85be0d740 1 -- 172.21.15.2:0/4123777455 shutdown_connections 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.675+0000 7fe85be0d740 1 -- 172.21.15.2:0/4123777455 wait complete. 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.675+0000 7fe852800640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 0x7fe8380a40d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe844009e10 tx=0x7fe844005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.675+0000 7fe85be0d740 1 -- start start 2024-03-11T18:42:55.680 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.675+0000 7fe85be0d740 1 Processor -- start 2024-03-11T18:42:55.684 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa0949f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa0949f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa093ff5d0 con 0x55aa09499b20 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa093a4b50 con 0x55aa094a2330 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa093a47f0 con 0x55aa09498db0 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe85be0d740 10 client.0 _ll_register_callbacks cb 0x55aa093b8550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe852800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe852800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:37256/0 (socket says 172.21.15.2:37256) 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe852800640 1 -- 172.21.15.2:0/3223472946 learned_addr learned my addr 172.21.15.2:0/3223472946 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe851400640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa0949f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.676+0000 7fe851e00640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa0949f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.685 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 -- 172.21.15.2:0/3223472946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 msgr2=0x55aa0949f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa0949f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 -- 172.21.15.2:0/3223472946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 msgr2=0x55aa0949f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa0949f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 -- 172.21.15.2:0/3223472946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85410e850 con 0x55aa094a2330 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe851400640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa0949f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe851e00640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa0949f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe852800640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe8440045f0 tx=0x7fe844004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924dddf4a318a7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe837e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844017070 con 0x55aa094a2330 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fe84401f640 con 0x55aa094a2330 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.677+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa093d1cd0 con 0x55aa094a2330 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe84401f800 con 0x55aa094a2330 2024-03-11T18:42:55.686 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 10 client.26213 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55aa093fdc50 con 0x55aa094a2330 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aa093d1cd0 con 0x55aa094a2330 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe844029a80 con 0x55aa094a2330 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 10 client.26213 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 20 client.26213 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa094b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 4 client.26213 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.678+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa093f5ce0 con 0x55aa094b2440 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.679+0000 7fe851e00640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa094b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.679+0000 7fe851e00640 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa094b4860 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48818719bd946ad7 server_cookie=3c7ee44f6b836ed5 in_seq=0 out_seq=0 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.679+0000 7fe837e00640 10 client.26213 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe837e00640 1 -- 172.21.15.2:0/3223472946 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aa093f5ce0 con 0x55aa094b2440 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe837e00640 10 client.26213 handle_command_reply: tid=0 2024-03-11T18:42:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 client.26213 shutdown 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 msgr2=0x55aa0949fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe8440045f0 tx=0x7fe844004730 comp rx=0 tx=0).stop 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 shutdown_connections 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa094b4860 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949fe40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe837e00640 0 client.26213 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09499b20 0x55aa0949f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 --2- 172.21.15.2:0/3223472946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09498db0 0x55aa0949f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.680+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 >> 172.21.15.2:0/3223472946 conn(0x55aa094604b0 msgr2=0x55aa094957f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.681+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 shutdown_connections 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.681+0000 7fe85be0d740 1 -- 172.21.15.2:0/3223472946 wait complete. 2024-03-11T18:42:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.681+0000 7fe85be0d740 20 client.26213 trim_cache size 0 max 16384 2024-03-11T18:42:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 Processor -- start 2024-03-11T18:42:55.701 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 -- start start 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa09147e60 con 0x55aa09498db0 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa0949a110 con 0x55aa094a2330 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.698+0000 7fe85be0d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa0949a2b0 con 0x55aa09499b20 2024-03-11T18:42:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe852800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38666/0 (socket says 172.21.15.2:38666) 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 -- 172.21.15.2:0/2991980995 learned_addr learned my addr 172.21.15.2:0/2991980995 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851400640 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 -- 172.21.15.2:0/2991980995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 msgr2=0x55aa0949fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 -- 172.21.15.2:0/2991980995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 msgr2=0x55aa094a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 -- 172.21.15.2:0/2991980995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa0939a530 con 0x55aa09498db0 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.699+0000 7fe851e00640 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009bf0 tx=0x7fe84c042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481648f23dc63c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.700+0000 7fe842a00640 1 -- 172.21.15.2:0/2991980995 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c056070 con 0x55aa09498db0 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.700+0000 7fe842a00640 1 -- 172.21.15.2:0/2991980995 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe84c052b70 con 0x55aa09498db0 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.700+0000 7fe842a00640 1 -- 172.21.15.2:0/2991980995 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84c003bd0 con 0x55aa09498db0 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.700+0000 7fe85be0d740 1 -- 172.21.15.2:0/2991980995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 msgr2=0x55aa094a1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.700+0000 7fe85be0d740 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009bf0 tx=0x7fe84c042040 comp rx=0 tx=0).stop 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 -- 172.21.15.2:0/2991980995 shutdown_connections 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa094a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 --2- 172.21.15.2:0/2991980995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a1af0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 -- 172.21.15.2:0/2991980995 >> 172.21.15.2:0/2991980995 conn(0x55aa094604b0 msgr2=0x55aa0949dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 -- 172.21.15.2:0/2991980995 shutdown_connections 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 -- 172.21.15.2:0/2991980995 wait complete. 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 -- start start 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.701+0000 7fe85be0d740 1 Processor -- start 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa094a9a50 con 0x55aa09498db0 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa0949c6e0 con 0x55aa094a2330 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa0949c8a0 con 0x55aa09499b20 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe85be0d740 10 client.0 _ll_register_callbacks cb 0x7fe8440044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe851400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe851e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe851400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:50832/0 (socket says 172.21.15.2:50832) 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe851400640 1 -- 172.21.15.2:0/1881964247 learned_addr learned my addr 172.21.15.2:0/1881964247 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 -- 172.21.15.2:0/1881964247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 msgr2=0x55aa0949bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 -- 172.21.15.2:0/1881964247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 msgr2=0x55aa094a92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe852800640 1 -- 172.21.15.2:0/1881964247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa09380b00 con 0x55aa094a2330 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.702+0000 7fe851400640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe852800640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949c170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe844019040 tx=0x7fe844029ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbebe6dc895765c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:42:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe837e00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:42:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844017040 con 0x55aa094a2330 2024-03-11T18:42:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa09424d30 con 0x55aa094a2330 2024-03-11T18:42:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fe844009e40 con 0x55aa094a2330 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe84401f850 con 0x55aa094a2330 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe85be0d740 10 client.26219 fetch_fsmap learned FSMap version 31 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.703+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55aa093c02c0 con 0x55aa094a2330 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aa09424d30 con 0x55aa094a2330 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe8440047a0 con 0x55aa094a2330 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 10 client.26219 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 10 client.26219 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 20 client.26219 populate_metadata read hostname 'smithi002' 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa0950bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 4 client.26219 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55aa090ff600 con 0x55aa094b2440 2024-03-11T18:42:55.710 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.704+0000 7fe851e00640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa0950bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:42:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.705+0000 7fe851e00640 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa0950bd20 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79e0ec97d417f133 server_cookie=9c37188fd9813568 in_seq=0 out_seq=0 2024-03-11T18:42:55.711 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.705+0000 7fe837e00640 10 client.26219 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe837e00640 1 -- 172.21.15.2:0/1881964247 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55aa090ff600 con 0x55aa094b2440 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe837e00640 10 client.26219 handle_command_reply: tid=0 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 client.26219 shutdown 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 msgr2=0x55aa0949c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949c170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe844019040 tx=0x7fe844029ca0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 shutdown_connections 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x55aa094b2440 0x55aa0950bd20 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aa094a2330 0x55aa0949c170 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55aa09499b20 0x55aa0949bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe85be0d740 1 --2- 172.21.15.2:0/1881964247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55aa09498db0 0x55aa094a92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.949+0000 7fe837e00640 0 client.26219 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:42:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.950+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 >> 172.21.15.2:0/1881964247 conn(0x55aa094604b0 msgr2=0x55aa094924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.950+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 shutdown_connections 2024-03-11T18:42:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.950+0000 7fe85be0d740 1 -- 172.21.15.2:0/1881964247 wait complete. 2024-03-11T18:42:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.950+0000 7fe85be0d740 20 client.26219 trim_cache size 0 max 16384 2024-03-11T18:42:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:42:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:42:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:42:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:42:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 msgr2=0x7fe8380a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 0x7fe8380a40d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe844009e10 tx=0x7fe844005f90 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 msgr2=0x7fe85419fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9f0 tx=0x7fe83c00eeb0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 shutdown_connections 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe854199880 0x7fe854199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe854106470 0x7fe85419fc10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe8380a1c10 0x7fe8380a40d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 --2- 172.21.15.2:0/1317159827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe8541013a0 0x7fe85419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.951+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 >> 172.21.15.2:0/1317159827 conn(0x7fe8540fcc00 msgr2=0x7fe8540fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.952+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 shutdown_connections 2024-03-11T18:42:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:42:55.952+0000 7fe859200640 1 -- 172.21.15.2:0/1317159827 wait complete. 2024-03-11T18:42:56.208 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:55 smithi023 ceph-mon[36406]: pgmap v287: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-03-11T18:42:56.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:55 smithi002 ceph-mon[31328]: pgmap v287: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-03-11T18:42:56.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:55 smithi165 ceph-mon[36634]: pgmap v287: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 318 op/s 2024-03-11T18:42:56.442 INFO:tasks.workunit.client.0.smithi002.stdout: 1 696953 7.47 MB/sec execute 356 sec latency 156.729 ms 2024-03-11T18:42:57.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 698457 7.47 MB/sec execute 357 sec latency 27.485 ms 2024-03-11T18:42:58.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:57 smithi002 ceph-mon[31328]: pgmap v288: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-03-11T18:42:58.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:57 smithi165 ceph-mon[36634]: pgmap v288: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-03-11T18:42:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:57 smithi023 ceph-mon[36406]: pgmap v288: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-03-11T18:42:58.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 700047 7.48 MB/sec execute 358 sec latency 64.975 ms 2024-03-11T18:42:59.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 701232 7.48 MB/sec execute 359 sec latency 27.489 ms 2024-03-11T18:43:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:42:59 smithi002 ceph-mon[31328]: pgmap v289: 129 pgs: 129 active+clean; 107 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-03-11T18:43:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:42:59 smithi165 ceph-mon[36634]: pgmap v289: 129 pgs: 129 active+clean; 107 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-03-11T18:43:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:42:59 smithi023 ceph-mon[36406]: pgmap v289: 129 pgs: 129 active+clean; 107 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-03-11T18:43:00.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 703235 7.49 MB/sec execute 360 sec latency 58.914 ms 2024-03-11T18:43:01.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704138 7.48 MB/sec execute 361 sec latency 23.195 ms 2024-03-11T18:43:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:01 smithi002 ceph-mon[31328]: pgmap v290: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 234 op/s 2024-03-11T18:43:02.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:01 smithi165 ceph-mon[36634]: pgmap v290: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 234 op/s 2024-03-11T18:43:02.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:01 smithi023 ceph-mon[36406]: pgmap v290: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 234 op/s 2024-03-11T18:43:02.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704147 7.46 MB/sec execute 362 sec latency 505.924 ms 2024-03-11T18:43:03.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:02 smithi002 ceph-mon[31328]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T18:43:03.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:02 smithi165 ceph-mon[36634]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T18:43:03.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:02 smithi023 ceph-mon[36406]: osdmap e105: 12 total, 12 up, 12 in 2024-03-11T18:43:03.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704158 7.44 MB/sec execute 363 sec latency 733.181 ms 2024-03-11T18:43:04.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:03 smithi002 ceph-mon[31328]: pgmap v292: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-03-11T18:43:04.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:03 smithi165 ceph-mon[36634]: pgmap v292: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-03-11T18:43:04.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:03 smithi023 ceph-mon[36406]: pgmap v292: 129 pgs: 129 active+clean; 102 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 256 op/s 2024-03-11T18:43:04.445 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704159 7.42 MB/sec execute 364 sec latency 857.616 ms 2024-03-11T18:43:05.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704161 7.40 MB/sec execute 365 sec latency 760.191 ms 2024-03-11T18:43:06.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:05 smithi002 ceph-mon[31328]: pgmap v293: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:06.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:05 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:43:06.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:05 smithi165 ceph-mon[36634]: pgmap v293: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:06.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:05 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:43:06.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:05 smithi023 ceph-mon[36406]: pgmap v293: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:06.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:05 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:43:06.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 704294 7.38 MB/sec execute 366 sec latency 284.720 ms 2024-03-11T18:43:07.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:06 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:43:07.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:06 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:43:07.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:06 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:43:07.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:06 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:43:07.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:06 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:43:07.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:06 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:43:07.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:06 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:43:07.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:06 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:43:07.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:06 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:43:07.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 705849 7.38 MB/sec execute 367 sec latency 34.814 ms 2024-03-11T18:43:08.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:07 smithi002 ceph-mon[31328]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:07 smithi165 ceph-mon[36634]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:08.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:07 smithi023 ceph-mon[36406]: pgmap v294: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2024-03-11T18:43:08.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 707238 7.38 MB/sec execute 368 sec latency 84.502 ms 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 -- 172.21.15.2:0/1561906674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc84103f80 msgr2=0x7fcc84107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 --2- 172.21.15.2:0/1561906674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc84103f80 0x7fcc84107fc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009860 tx=0x7fcc7402bb70 comp rx=0 tx=0).stop 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 -- 172.21.15.2:0/1561906674 shutdown_connections 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 --2- 172.21.15.2:0/1561906674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841086a0 0x7fcc84110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 --2- 172.21.15.2:0/1561906674 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc84103f80 0x7fcc84107fc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 --2- 172.21.15.2:0/1561906674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841035d0 0x7fcc841039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.100+0000 7fcc8c800640 1 -- 172.21.15.2:0/1561906674 >> 172.21.15.2:0/1561906674 conn(0x7fcc840fee70 msgr2=0x7fcc84101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.101+0000 7fcc8c800640 1 -- 172.21.15.2:0/1561906674 shutdown_connections 2024-03-11T18:43:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.101+0000 7fcc8c800640 1 -- 172.21.15.2:0/1561906674 wait complete. 2024-03-11T18:43:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.102+0000 7fcc8c800640 1 Processor -- start 2024-03-11T18:43:09.104 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.102+0000 7fcc8c800640 1 -- start start 2024-03-11T18:43:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841086a0 0x7fcc8419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 0x7fcc84199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419a350 con 0x7fcc841086a0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419a4c0 con 0x7fcc841998a0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8c800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419a630 con 0x7fcc841035d0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 0x7fcc84199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41474/0 (socket says 172.21.15.2:41474) 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc89e00640 1 -- 172.21.15.2:0/2914361815 learned_addr learned my addr 172.21.15.2:0/2914361815 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc89400640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841086a0 0x7fcc8419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 -- 172.21.15.2:0/2914361815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 msgr2=0x7fcc8419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 -- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841086a0 msgr2=0x7fcc8419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841086a0 0x7fcc8419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc8a800640 1 -- 172.21.15.2:0/2914361815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78009560 con 0x7fcc841998a0 2024-03-11T18:43:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.103+0000 7fcc89e00640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc8a800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 0x7fcc84199d50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009fd0 tx=0x7fcc8000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc7ea00640 1 -- 172.21.15.2:0/2914361815 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000ce20 con 0x7fcc841998a0 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc74009490 con 0x7fcc841998a0 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc7ea00640 1 -- 172.21.15.2:0/2914361815 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc80005020 con 0x7fcc841998a0 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc7ea00640 1 -- 172.21.15.2:0/2914361815 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80010690 con 0x7fcc841998a0 2024-03-11T18:43:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.104+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc841a6890 con 0x7fcc841998a0 2024-03-11T18:43:09.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.107+0000 7fcc7ea00640 1 -- 172.21.15.2:0/2914361815 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fcc80005500 con 0x7fcc841998a0 2024-03-11T18:43:09.109 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.107+0000 7fcc8f3f3740 1 Processor -- start 2024-03-11T18:43:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 -- start start 2024-03-11T18:43:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 0x5647e9d91870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9d9d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9d4d620 con 0x5647e9d91db0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9cff8c0 con 0x5647e9d92b20 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9cfcdf0 con 0x5647e9d9b330 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43580/0 (socket says 172.21.15.2:43580) 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc89e00640 1 -- 172.21.15.2:0/1118410652 learned_addr learned my addr 172.21.15.2:0/1118410652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.108+0000 7fcc89400640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 0x5647e9d91870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89400640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9d9d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 -- 172.21.15.2:0/1118410652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 msgr2=0x5647e9d9d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9d9d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 -- 172.21.15.2:0/1118410652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 msgr2=0x5647e9d91870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 0x5647e9d91870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 -- 172.21.15.2:0/1118410652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e9c79b00 con 0x5647e9d91db0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89400640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 0x5647e9d91870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc7ea00640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 0x7fcc6c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc89e00640 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcc78011810 tx=0x7fcc78011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc9979144d7b1f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.109+0000 7fcc7ea00640 1 -- 172.21.15.2:0/2914361815 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fcc80014070 con 0x7fcc841998a0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc73000640 1 -- 172.21.15.2:0/1118410652 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7800fba0 con 0x5647e9d91db0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc89400640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 0x7fcc6c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc73000640 1 -- 172.21.15.2:0/1118410652 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc7800fd00 con 0x5647e9d91db0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc73000640 1 -- 172.21.15.2:0/1118410652 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7800e5f0 con 0x5647e9d91db0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/1118410652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 msgr2=0x5647e9d92230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc89400640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 0x7fcc6c0a4050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009830 tx=0x7fcc74000bc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.110+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcc78011810 tx=0x7fcc78011c10 comp rx=0 tx=0).stop 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/1118410652 shutdown_connections 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9d9d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d91db0 0x5647e9d92230 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/1118410652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d92b20 0x5647e9d91870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/1118410652 >> 172.21.15.2:0/1118410652 conn(0x5647e9d594b0 msgr2=0x5647e9d8caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/1118410652 shutdown_connections 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/1118410652 wait complete. 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 -- start start 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.111+0000 7fcc8f3f3740 1 Processor -- start 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d988a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9d98de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9cf85d0 con 0x5647e9d9b330 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9c9db50 con 0x5647e9d91db0 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9c9d7f0 con 0x5647e9d92b20 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8f3f3740 10 client.0 _ll_register_callbacks cb 0x5647e9cb1550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60270/0 (socket says 172.21.15.2:60270) 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc89e00640 1 -- 172.21.15.2:0/633962652 learned_addr learned my addr 172.21.15.2:0/633962652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc89400640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.112+0000 7fcc8a800640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9d98de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 -- 172.21.15.2:0/633962652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 msgr2=0x5647e9d988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 -- 172.21.15.2:0/633962652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 msgr2=0x5647e9d98de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9d98de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 -- 172.21.15.2:0/633962652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc78011440 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc89e00640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcc7801c040 tx=0x7fcc78005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9ddaaeac8c46c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc67400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78004230 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fcc78004390 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.113+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647e9ccacd0 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc78005670 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 10 client.26231 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5647e9cf6c50 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5647e9ccacd0 con 0x5647e9d91db0 2024-03-11T18:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcc7802c060 con 0x5647e9d91db0 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 10 client.26231 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 20 client.26231 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9dab270 0x5647e9dad660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 4 client.26231 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647e9ceece0 con 0x5647e9dab270 2024-03-11T18:43:09.120 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.114+0000 7fcc89400640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9dab270 0x5647e9dad660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.115+0000 7fcc89400640 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9dab270 0x5647e9dad660 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434ab82d2fd98fa1 server_cookie=86b0eb2a777d0f7d in_seq=0 out_seq=0 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.115+0000 7fcc67400640 10 client.26231 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc67400640 1 -- 172.21.15.2:0/633962652 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5647e9ceece0 con 0x5647e9dab270 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc67400640 10 client.26231 handle_command_reply: tid=0 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 client.26231 shutdown 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 msgr2=0x5647e9d98360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fcc7801c040 tx=0x7fcc78005270 comp rx=0 tx=0).stop 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 shutdown_connections 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9dab270 0x5647e9dad660 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9d98de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc67400640 0 client.26231 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/633962652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d98360 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.117+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 >> 172.21.15.2:0/633962652 conn(0x5647e9d594b0 msgr2=0x5647e9d8e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.118+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 shutdown_connections 2024-03-11T18:43:09.121 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.118+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/633962652 wait complete. 2024-03-11T18:43:09.122 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.118+0000 7fcc8f3f3740 20 client.26231 trim_cache size 0 max 16384 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 Processor -- start 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 -- start start 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d92b20 0x5647e9da2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9da2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9a40e60 con 0x5647e9d92b20 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9d88370 con 0x5647e9d91db0 2024-03-11T18:43:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.135+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9d91080 con 0x5647e9d9b330 2024-03-11T18:43:09.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.138 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc8a800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9da2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d92b20 0x5647e9da2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60272/0 (socket says 172.21.15.2:60272) 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 -- 172.21.15.2:0/2412976688 learned_addr learned my addr 172.21.15.2:0/2412976688 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 -- 172.21.15.2:0/2412976688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 msgr2=0x5647e9da2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9da2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 -- 172.21.15.2:0/2412976688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d92b20 msgr2=0x5647e9da2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d92b20 0x5647e9da2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 -- 172.21.15.2:0/2412976688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e9c93530 con 0x5647e9d91db0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc8a800640 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9da2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.136+0000 7fcc89e00640 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcc78019d30 tx=0x7fcc78005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=875f8ba80617acf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc72600640 1 -- 172.21.15.2:0/2412976688 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7800f840 con 0x5647e9d91db0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc72600640 1 -- 172.21.15.2:0/2412976688 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc78005970 con 0x5647e9d91db0 2024-03-11T18:43:09.139 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc72600640 1 -- 172.21.15.2:0/2412976688 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc780039f0 con 0x5647e9d91db0 2024-03-11T18:43:09.141 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/2412976688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 msgr2=0x5647e9d9aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcc78019d30 tx=0x7fcc78005df0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/2412976688 shutdown_connections 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d9b330 0x5647e9da2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d92b20 0x5647e9da2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/2412976688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9aa90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.137+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/2412976688 >> 172.21.15.2:0/2412976688 conn(0x5647e9d594b0 msgr2=0x5647e9d912d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.138+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/2412976688 shutdown_connections 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.138+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/2412976688 wait complete. 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.138+0000 7fcc8f3f3740 1 -- start start 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.138+0000 7fcc8f3f3740 1 Processor -- start 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9e00720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.142 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9d9ef30 con 0x5647e9d9b330 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9e00c60 con 0x5647e9d91db0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e9e00da0 con 0x5647e9d92b20 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8f3f3740 10 client.0 _ll_register_callbacks cb 0x7fcc7801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:41520/0 (socket says 172.21.15.2:41520) 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 -- 172.21.15.2:0/3164715777 learned_addr learned my addr 172.21.15.2:0/3164715777 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8a800640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9e00720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 -- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 msgr2=0x5647e9d9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 -- 172.21.15.2:0/3164715777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 msgr2=0x5647e9e00720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9e00720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc89400640 1 -- 172.21.15.2:0/3164715777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc841a6890 con 0x5647e9d92b20 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.139+0000 7fcc8a800640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9e00720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc89400640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcc7402c180 tx=0x7fcc74005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64cdd641f82564b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:09.143 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc67400640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74054040 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fcc74048040 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647e9d1dd30 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc740347c0 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc8f3f3740 10 client.26233 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.140+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5647e9cf6c50 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5647e9d1dd30 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcc74032930 con 0x5647e9d92b20 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 10 client.26233 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 10 client.26233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 20 client.26233 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9e02550 0x5647e9e04970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 4 client.26233 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:43:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5647e99f8600 con 0x5647e9e02550 2024-03-11T18:43:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.141+0000 7fcc89e00640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9e02550 0x5647e9e04970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.142+0000 7fcc67400640 10 client.26233 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:09.146 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.142+0000 7fcc89e00640 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9e02550 0x5647e9e04970 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853e4f6266a46c8b server_cookie=cda2392cf429f16d in_seq=0 out_seq=0 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.366+0000 7fcc67400640 1 -- 172.21.15.2:0/3164715777 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5647e99f8600 con 0x5647e9e02550 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.366+0000 7fcc67400640 10 client.26233 handle_command_reply: tid=0 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.366+0000 7fcc8f3f3740 1 client.26233 shutdown 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 msgr2=0x5647e9d9e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcc7402c180 tx=0x7fcc74005f90 comp rx=0 tx=0).stop 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 shutdown_connections 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5647e9e02550 0x5647e9e04970 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5647e9d9b330 0x5647e9e00720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc67400640 0 client.26233 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5647e9d92b20 0x5647e9d9e860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 --2- 172.21.15.2:0/3164715777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e9d91db0 0x5647e9d9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 >> 172.21.15.2:0/3164715777 conn(0x5647e9d594b0 msgr2=0x5647e9d96b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 shutdown_connections 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 1 -- 172.21.15.2:0/3164715777 wait complete. 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.367+0000 7fcc8f3f3740 20 client.26233 trim_cache size 0 max 16384 2024-03-11T18:43:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 msgr2=0x7fcc6c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 0x7fcc6c0a4050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009830 tx=0x7fcc74000bc0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 msgr2=0x7fcc84199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 0x7fcc84199d50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc80009fd0 tx=0x7fcc8000ef10 comp rx=0 tx=0).stop 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 shutdown_connections 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc841998a0 0x7fcc84199d50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcc6c0a1b90 0x7fcc6c0a4050 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc841086a0 0x7fcc8419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 --2- 172.21.15.2:0/2914361815 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc841035d0 0x7fcc8419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 >> 172.21.15.2:0/2914361815 conn(0x7fcc840fee70 msgr2=0x7fcc84100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 shutdown_connections 2024-03-11T18:43:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:09.368+0000 7fcc8c800640 1 -- 172.21.15.2:0/2914361815 wait complete. 2024-03-11T18:43:09.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 708232 7.38 MB/sec execute 369 sec latency 82.483 ms 2024-03-11T18:43:10.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:09 smithi002 ceph-mon[31328]: pgmap v295: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 104 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.7 MiB/s wr, 172 op/s 2024-03-11T18:43:10.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:09 smithi165 ceph-mon[36634]: pgmap v295: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 104 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.7 MiB/s wr, 172 op/s 2024-03-11T18:43:10.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:09 smithi023 ceph-mon[36406]: pgmap v295: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 104 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.7 MiB/s wr, 172 op/s 2024-03-11T18:43:10.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 710042 7.38 MB/sec execute 370 sec latency 30.093 ms 2024-03-11T18:43:11.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 711092 7.38 MB/sec execute 371 sec latency 23.690 ms 2024-03-11T18:43:12.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:11 smithi002 ceph-mon[31328]: pgmap v296: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-03-11T18:43:12.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:11 smithi165 ceph-mon[36634]: pgmap v296: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-03-11T18:43:12.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:11 smithi023 ceph-mon[36406]: pgmap v296: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 1.7 MiB/s wr, 167 op/s 2024-03-11T18:43:12.446 INFO:tasks.workunit.client.0.smithi002.stdout: 1 712365 7.37 MB/sec execute 372 sec latency 50.453 ms 2024-03-11T18:43:13.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 713924 7.38 MB/sec execute 373 sec latency 50.339 ms 2024-03-11T18:43:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:13 smithi002 ceph-mon[31328]: pgmap v297: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-03-11T18:43:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:13 smithi002 ceph-mon[31328]: osdmap e106: 12 total, 12 up, 12 in 2024-03-11T18:43:14.405 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:13 smithi165 ceph-mon[36634]: pgmap v297: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-03-11T18:43:14.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:13 smithi165 ceph-mon[36634]: osdmap e106: 12 total, 12 up, 12 in 2024-03-11T18:43:14.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:13 smithi023 ceph-mon[36406]: pgmap v297: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.6 MiB/s wr, 156 op/s 2024-03-11T18:43:14.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:13 smithi023 ceph-mon[36406]: osdmap e106: 12 total, 12 up, 12 in 2024-03-11T18:43:14.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 714667 7.37 MB/sec execute 374 sec latency 25.570 ms 2024-03-11T18:43:15.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 716112 7.37 MB/sec execute 375 sec latency 27.200 ms 2024-03-11T18:43:16.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:16 smithi023 ceph-mon[36406]: pgmap v299: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:16.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:16 smithi165 ceph-mon[36634]: pgmap v299: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:16.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 717652 7.38 MB/sec execute 376 sec latency 15.387 ms 2024-03-11T18:43:16.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:16 smithi002 ceph-mon[31328]: pgmap v299: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:17.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 718723 7.37 MB/sec execute 377 sec latency 26.714 ms 2024-03-11T18:43:18.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:18 smithi165 ceph-mon[36634]: pgmap v300: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:18.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:18 smithi023 ceph-mon[36406]: pgmap v300: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:18.447 INFO:tasks.workunit.client.0.smithi002.stdout: 1 720191 7.37 MB/sec execute 378 sec latency 76.966 ms 2024-03-11T18:43:18.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:18 smithi002 ceph-mon[31328]: pgmap v300: 129 pgs: 129 active+clean; 102 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2024-03-11T18:43:19.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 721409 7.37 MB/sec execute 379 sec latency 26.605 ms 2024-03-11T18:43:20.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:20 smithi165 ceph-mon[36634]: pgmap v301: 129 pgs: 129 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-03-11T18:43:20.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:20 smithi023 ceph-mon[36406]: pgmap v301: 129 pgs: 129 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-03-11T18:43:20.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 722505 7.37 MB/sec execute 380 sec latency 29.329 ms 2024-03-11T18:43:20.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:20 smithi002 ceph-mon[31328]: pgmap v301: 129 pgs: 129 active+clean; 106 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 273 op/s 2024-03-11T18:43:21.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 724029 7.37 MB/sec execute 381 sec latency 33.404 ms 2024-03-11T18:43:22.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:22 smithi165 ceph-mon[36634]: pgmap v302: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:22.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:22 smithi023 ceph-mon[36406]: pgmap v302: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:22.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 724977 7.36 MB/sec execute 382 sec latency 15.496 ms 2024-03-11T18:43:22.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:22 smithi002 ceph-mon[31328]: pgmap v302: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 -- 172.21.15.2:0/669203515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c106460 msgr2=0x7f094c10e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 --2- 172.21.15.2:0/669203515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c106460 0x7f094c10e030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0944009ca0 tx=0x7f094402c100 comp rx=0 tx=0).stop 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 -- 172.21.15.2:0/669203515 shutdown_connections 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 --2- 172.21.15.2:0/669203515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c106460 0x7f094c10e030 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 --2- 172.21.15.2:0/669203515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c069640 0x7f094c105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 --2- 172.21.15.2:0/669203515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c069070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 -- 172.21.15.2:0/669203515 >> 172.21.15.2:0/669203515 conn(0x7f094c077170 msgr2=0x7f094c077580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 -- 172.21.15.2:0/669203515 shutdown_connections 2024-03-11T18:43:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.522+0000 7f0953800640 1 -- 172.21.15.2:0/669203515 wait complete. 2024-03-11T18:43:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.523+0000 7f0953800640 1 Processor -- start 2024-03-11T18:43:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.523+0000 7f0953800640 1 -- start start 2024-03-11T18:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.523+0000 7f0953800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0953800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 0x7f094c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0953800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 0x7f094c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0953800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c197f80 con 0x7f094c068c90 2024-03-11T18:43:22.527 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0953800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1980f0 con 0x7f094c106460 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0953800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094c1982b0 con 0x7f094c197590 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0950e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 0x7f094c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0950e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46912/0 (socket says 172.21.15.2:46912) 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0950e00640 1 -- 172.21.15.2:0/2264334918 learned_addr learned my addr 172.21.15.2:0/2264334918 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0951800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 0x7f094c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 -- 172.21.15.2:0/2264334918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 msgr2=0x7f094c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 0x7f094c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 -- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 msgr2=0x7f094c19d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f094be00640 1 -- 172.21.15.2:0/2264334918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938009490 con 0x7f094c106460 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0950e00640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.524+0000 7f0951800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 0x7f094c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f094be00640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 0x7f094c19d920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f093800e9c0 tx=0x7f093800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.528 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f0949600640 1 -- 172.21.15.2:0/2264334918 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093800cd80 con 0x7f094c106460 2024-03-11T18:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f0949600640 1 -- 172.21.15.2:0/2264334918 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09380035f0 con 0x7f094c106460 2024-03-11T18:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0944009900 con 0x7f094c106460 2024-03-11T18:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f0949600640 1 -- 172.21.15.2:0/2264334918 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938010670 con 0x7f094c106460 2024-03-11T18:43:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.525+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c1a4620 con 0x7f094c106460 2024-03-11T18:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.529+0000 7f0956582740 1 Processor -- start 2024-03-11T18:43:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.529+0000 7f0949600640 1 -- 172.21.15.2:0/2264334918 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f0938003af0 con 0x7f094c106460 2024-03-11T18:43:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 -- start start 2024-03-11T18:43:22.532 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90cf30 0x5577cd90bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd917b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd8c3620 con 0x5577cd90cf30 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd8758c0 con 0x5577cd90c1c0 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0956582740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd872df0 con 0x5577cd915740 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0950e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f094be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd917b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0951800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90cf30 0x5577cd90bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0950e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45606/0 (socket says 172.21.15.2:45606) 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.530+0000 7f0950e00640 1 -- 172.21.15.2:0/3270022816 learned_addr learned my addr 172.21.15.2:0/3270022816 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 -- 172.21.15.2:0/3270022816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 msgr2=0x5577cd917b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd917b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 -- 172.21.15.2:0/3270022816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90cf30 msgr2=0x5577cd90bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90cf30 0x5577cd90bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 -- 172.21.15.2:0/3270022816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577cd7efb00 con 0x5577cd90c1c0 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0950e00640 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0940011650 tx=0x7f0940009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e8a46b9362ad61b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f093e000640 1 -- 172.21.15.2:0/3270022816 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000f6c0 con 0x5577cd90c1c0 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f093e000640 1 -- 172.21.15.2:0/3270022816 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f094000fd00 con 0x5577cd90c1c0 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f093e000640 1 -- 172.21.15.2:0/3270022816 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940019e00 con 0x5577cd90c1c0 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.531+0000 7f0949600640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 0x7f09340a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0949600640 1 -- 172.21.15.2:0/2264334918 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f0938014070 con 0x7f094c106460 2024-03-11T18:43:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0951800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 0x7f09340a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0956582740 1 -- 172.21.15.2:0/3270022816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 msgr2=0x5577cd90c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0956582740 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0940011650 tx=0x7f0940009f90 comp rx=0 tx=0).stop 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0956582740 1 -- 172.21.15.2:0/3270022816 shutdown_connections 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0956582740 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd917b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.532+0000 7f0956582740 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd90c640 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 --2- 172.21.15.2:0/3270022816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90cf30 0x5577cd90bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 -- 172.21.15.2:0/3270022816 >> 172.21.15.2:0/3270022816 conn(0x5577cd8cf4b0 msgr2=0x5577cd906eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 -- 172.21.15.2:0/3270022816 shutdown_connections 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0951800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 0x7f09340a4050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0944004af0 tx=0x7f0944005c50 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 -- 172.21.15.2:0/3270022816 wait complete. 2024-03-11T18:43:22.536 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 -- start start 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.533+0000 7f0956582740 1 Processor -- start 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd912cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd915740 0x5577cd913230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd86e5d0 con 0x5577cd915740 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd813b50 con 0x5577cd90c1c0 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd8137f0 con 0x5577cd90cf30 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0956582740 10 client.0 _ll_register_callbacks cb 0x5577cd827550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0950e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0950e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45612/0 (socket says 172.21.15.2:45612) 2024-03-11T18:43:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0950e00640 1 -- 172.21.15.2:0/3106052858 learned_addr learned my addr 172.21.15.2:0/3106052858 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f094be00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd912cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.534+0000 7f0951800640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd915740 0x5577cd913230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 -- 172.21.15.2:0/3106052858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 msgr2=0x5577cd912cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd912cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 -- 172.21.15.2:0/3106052858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd915740 msgr2=0x5577cd913230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd915740 0x5577cd913230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 -- 172.21.15.2:0/3106052858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0940009e00 con 0x5577cd90c1c0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f094be00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd912cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0950e00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0940019640 tx=0x7f0940019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5edc185d8c188d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.535+0000 7f0932a00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940004240 con 0x5577cd90c1c0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f09400043a0 con 0x5577cd90c1c0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577cd840cd0 con 0x5577cd90c1c0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0940005040 con 0x5577cd90c1c0 2024-03-11T18:43:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0956582740 10 client.26249 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5577cd86cc50 con 0x5577cd90c1c0 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5577cd840cd0 con 0x5577cd90c1c0 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.536+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f094000fbf0 con 0x5577cd90c1c0 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 10 client.26249 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 20 client.26249 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd925730 0x5577cd927b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 4 client.26249 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577cd864ce0 con 0x5577cd925730 2024-03-11T18:43:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.537+0000 7f094be00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd925730 0x5577cd927b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.538+0000 7f094be00640 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd925730 0x5577cd927b20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=625ec7b56f990677 server_cookie=8b0ed1e0c7b0e693 in_seq=0 out_seq=0 2024-03-11T18:43:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.538+0000 7f0932a00640 10 client.26249 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.538+0000 7f0932a00640 1 -- 172.21.15.2:0/3106052858 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5577cd864ce0 con 0x5577cd925730 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.538+0000 7f0932a00640 10 client.26249 handle_command_reply: tid=0 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.538+0000 7f0956582740 1 client.26249 shutdown 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 msgr2=0x5577cd9127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0940019640 tx=0x7f0940019670 comp rx=0 tx=0).stop 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 shutdown_connections 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd925730 0x5577cd927b20 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd915740 0x5577cd913230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd912cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 --2- 172.21.15.2:0/3106052858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90c1c0 0x5577cd9127b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0932a00640 0 client.26249 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 >> 172.21.15.2:0/3106052858 conn(0x5577cd8cf4b0 msgr2=0x5577cd908370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 shutdown_connections 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 1 -- 172.21.15.2:0/3106052858 wait complete. 2024-03-11T18:43:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.539+0000 7f0956582740 20 client.26249 trim_cache size 0 max 16384 2024-03-11T18:43:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.560+0000 7f0956582740 1 Processor -- start 2024-03-11T18:43:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.560+0000 7f0956582740 1 -- start start 2024-03-11T18:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd914ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd91cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd5b6e60 con 0x5577cd90c1c0 2024-03-11T18:43:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd925530 con 0x5577cd90cf30 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0956582740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd908110 con 0x5577cd915740 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f094be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f094be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45624/0 (socket says 172.21.15.2:45624) 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f094be00640 1 -- 172.21.15.2:0/407622372 learned_addr learned my addr 172.21.15.2:0/407622372 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0950e00640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd914ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.561+0000 7f0951800640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd91cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 -- 172.21.15.2:0/407622372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 msgr2=0x5577cd91cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd91cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 -- 172.21.15.2:0/407622372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 msgr2=0x5577cd914ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd914ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 -- 172.21.15.2:0/407622372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577cd809530 con 0x5577cd90cf30 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f0951800640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd91cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f0950e00640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd914ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f094be00640 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f093800c6a0 tx=0x7f09380d02a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f54a9b7e17dadf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f093d600640 1 -- 172.21.15.2:0/407622372 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09380b8ea0 con 0x5577cd90cf30 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f093d600640 1 -- 172.21.15.2:0/407622372 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f093809a270 con 0x5577cd90cf30 2024-03-11T18:43:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.562+0000 7f093d600640 1 -- 172.21.15.2:0/407622372 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938098a30 con 0x5577cd90cf30 2024-03-11T18:43:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 -- 172.21.15.2:0/407622372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 msgr2=0x5577cd91c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f093800c6a0 tx=0x7f09380d02a0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 -- 172.21.15.2:0/407622372 shutdown_connections 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd915740 0x5577cd91cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd90cf30 0x5577cd91c570 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 --2- 172.21.15.2:0/407622372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd914ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 -- 172.21.15.2:0/407622372 >> 172.21.15.2:0/407622372 conn(0x5577cd8cf4b0 msgr2=0x5577cd92c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 -- 172.21.15.2:0/407622372 shutdown_connections 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.563+0000 7f0956582740 1 -- 172.21.15.2:0/407622372 wait complete. 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.564+0000 7f0956582740 1 -- start start 2024-03-11T18:43:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.564+0000 7f0956582740 1 Processor -- start 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd91c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd91afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd91bb80 con 0x5577cd90c1c0 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd91bcf0 con 0x5577cd915740 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577cd980420 con 0x5577cd90cf30 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0956582740 10 client.0 _ll_register_callbacks cb 0x7f094001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f094be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd91afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0951800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0951800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:45634/0 (socket says 172.21.15.2:45634) 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0950e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd91c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f0951800640 1 -- 172.21.15.2:0/419471760 learned_addr learned my addr 172.21.15.2:0/419471760 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.565+0000 7f094be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd91afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:58568/0 (socket says 172.21.15.2:58568) 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 -- 172.21.15.2:0/419471760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 msgr2=0x5577cd91afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd91afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 -- 172.21.15.2:0/419471760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 msgr2=0x5577cd91c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd91c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 -- 172.21.15.2:0/419471760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094c10e840 con 0x5577cd915740 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0951800640 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f09440095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c0d107d6b648dc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.566+0000 7f0932a00640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944051070 con 0x5577cd915740 2024-03-11T18:43:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577cd893d30 con 0x5577cd915740 2024-03-11T18:43:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f0944004250 con 0x5577cd915740 2024-03-11T18:43:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0944004410 con 0x5577cd915740 2024-03-11T18:43:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0956582740 10 client.26255 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f094c1a4620 con 0x5577cd915740 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.567+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5577cd893d30 con 0x5577cd915740 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0944048040 con 0x5577cd915740 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 10 client.26255 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 10 client.26255 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 20 client.26255 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd981af0 0x5577cd983f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 4 client.26255 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5577cd56e600 con 0x5577cd981af0 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.568+0000 7f0950e00640 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd981af0 0x5577cd983f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.569+0000 7f0950e00640 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd981af0 0x5577cd983f10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afba04c3fc386e19 server_cookie=7a3c2a701f202c3 in_seq=0 out_seq=0 2024-03-11T18:43:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.569+0000 7f0932a00640 10 client.26255 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.854+0000 7f0932a00640 1 -- 172.21.15.2:0/419471760 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5577cd56e600 con 0x5577cd981af0 2024-03-11T18:43:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.854+0000 7f0932a00640 10 client.26255 handle_command_reply: tid=0 2024-03-11T18:43:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.854+0000 7f0956582740 1 client.26255 shutdown 2024-03-11T18:43:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.854+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 msgr2=0x5577cd91b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.854+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f09440095c0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 shutdown_connections 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x5577cd981af0 0x5577cd983f10 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5577cd915740 0x5577cd91b4e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x5577cd90cf30 0x5577cd91afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 --2- 172.21.15.2:0/419471760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5577cd90c1c0 0x5577cd91c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0932a00640 0 client.26255 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 >> 172.21.15.2:0/419471760 conn(0x5577cd8cf4b0 msgr2=0x5577cd9069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 shutdown_connections 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 1 -- 172.21.15.2:0/419471760 wait complete. 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.855+0000 7f0956582740 20 client.26255 trim_cache size 0 max 16384 2024-03-11T18:43:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 msgr2=0x7f09340a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 0x7f09340a4050 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0944004af0 tx=0x7f0944005c50 comp rx=0 tx=0).stop 2024-03-11T18:43:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 msgr2=0x7f094c19d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 0x7f094c19d920 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f093800e9c0 tx=0x7f093800ee80 comp rx=0 tx=0).stop 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 shutdown_connections 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f094c197590 0x7f094c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f094c106460 0x7f094c19d920 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f09340a1b90 0x7f09340a4050 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 --2- 172.21.15.2:0/2264334918 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f094c068c90 0x7f094c19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 >> 172.21.15.2:0/2264334918 conn(0x7f094c077170 msgr2=0x7f094c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 shutdown_connections 2024-03-11T18:43:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:22.856+0000 7f0953800640 1 -- 172.21.15.2:0/2264334918 wait complete. 2024-03-11T18:43:23.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 725857 7.36 MB/sec execute 383 sec latency 168.718 ms 2024-03-11T18:43:24.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:24 smithi165 ceph-mon[36634]: pgmap v303: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:24.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:24 smithi023 ceph-mon[36406]: pgmap v303: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:24.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 727182 7.35 MB/sec execute 384 sec latency 51.008 ms 2024-03-11T18:43:24.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:24 smithi002 ceph-mon[31328]: pgmap v303: 129 pgs: 129 active+clean; 107 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 259 op/s 2024-03-11T18:43:25.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 728349 7.36 MB/sec execute 385 sec latency 17.583 ms 2024-03-11T18:43:26.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:26 smithi165 ceph-mon[36634]: pgmap v304: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-03-11T18:43:26.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:26 smithi023 ceph-mon[36406]: pgmap v304: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-03-11T18:43:26.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 729335 7.35 MB/sec execute 386 sec latency 28.223 ms 2024-03-11T18:43:26.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:26 smithi002 ceph-mon[31328]: pgmap v304: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-03-11T18:43:27.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 730382 7.34 MB/sec execute 387 sec latency 46.320 ms 2024-03-11T18:43:28.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:28 smithi165 ceph-mon[36634]: pgmap v305: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-03-11T18:43:28.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:28 smithi023 ceph-mon[36406]: pgmap v305: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-03-11T18:43:28.448 INFO:tasks.workunit.client.0.smithi002.stdout: 1 732016 7.35 MB/sec execute 388 sec latency 19.412 ms 2024-03-11T18:43:28.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:28 smithi002 ceph-mon[31328]: pgmap v305: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 227 op/s 2024-03-11T18:43:29.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 733104 7.34 MB/sec execute 389 sec latency 31.325 ms 2024-03-11T18:43:30.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:30 smithi165 ceph-mon[36634]: pgmap v306: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 277 op/s 2024-03-11T18:43:30.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:30 smithi023 ceph-mon[36406]: pgmap v306: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 277 op/s 2024-03-11T18:43:30.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 735021 7.36 MB/sec execute 390 sec latency 24.863 ms 2024-03-11T18:43:30.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:30 smithi002 ceph-mon[31328]: pgmap v306: 129 pgs: 129 active+clean; 100 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 277 op/s 2024-03-11T18:43:31.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 735951 7.34 MB/sec execute 391 sec latency 21.197 ms 2024-03-11T18:43:32.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:32 smithi165 ceph-mon[36634]: pgmap v307: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 245 op/s 2024-03-11T18:43:32.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:32 smithi023 ceph-mon[36406]: pgmap v307: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 245 op/s 2024-03-11T18:43:32.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 736813 7.34 MB/sec execute 392 sec latency 226.474 ms 2024-03-11T18:43:32.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:32 smithi002 ceph-mon[31328]: pgmap v307: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 245 op/s 2024-03-11T18:43:33.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 738394 7.34 MB/sec execute 393 sec latency 57.215 ms 2024-03-11T18:43:34.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:34 smithi165 ceph-mon[36634]: pgmap v308: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-03-11T18:43:34.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:34 smithi023 ceph-mon[36406]: pgmap v308: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-03-11T18:43:34.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 739309 7.33 MB/sec execute 394 sec latency 24.138 ms 2024-03-11T18:43:34.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:34 smithi002 ceph-mon[31328]: pgmap v308: 129 pgs: 129 active+clean; 102 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 233 op/s 2024-03-11T18:43:35.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 740235 7.33 MB/sec execute 395 sec latency 30.307 ms 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 -- 172.21.15.2:0/63839998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 msgr2=0x7f6b2c0692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 --2- 172.21.15.2:0/63839998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c0692f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009860 tx=0x7f6b1c02bc00 comp rx=0 tx=0).stop 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 -- 172.21.15.2:0/63839998 shutdown_connections 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 --2- 172.21.15.2:0/63839998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c10d770 0x7f6b2c10fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 --2- 172.21.15.2:0/63839998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c0692f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 --2- 172.21.15.2:0/63839998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c101f10 0x7f6b2c068970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 -- 172.21.15.2:0/63839998 >> 172.21.15.2:0/63839998 conn(0x7f6b2c0759e0 msgr2=0x7f6b2c075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 -- 172.21.15.2:0/63839998 shutdown_connections 2024-03-11T18:43:36.022 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.019+0000 7f6b30e00640 1 -- 172.21.15.2:0/63839998 wait complete. 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.020+0000 7f6b30e00640 1 Processor -- start 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 -- start start 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c101f10 0x7f6b2c19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 0x7f6b2c1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c199bf0 con 0x7f6b2c10d770 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c199d60 con 0x7f6b2c068eb0 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b30e00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c199f20 con 0x7f6b2c101f10 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55330/0 (socket says 172.21.15.2:55330) 2024-03-11T18:43:36.025 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b2b400640 1 -- 172.21.15.2:0/1506818537 learned_addr learned my addr 172.21.15.2:0/1506818537 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c101f10 0x7f6b2c19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.021+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 0x7f6b2c1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 -- 172.21.15.2:0/1506818537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c101f10 msgr2=0x7f6b2c19f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c101f10 0x7f6b2c19f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 -- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 msgr2=0x7f6b2c1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 0x7f6b2c1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 -- 172.21.15.2:0/1506818537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1c009490 con 0x7f6b2c068eb0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 0x7f6b2c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b2b400640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400e9f0 tx=0x7f6b1400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b23e00640 1 -- 172.21.15.2:0/1506818537 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14009700 con 0x7f6b2c068eb0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b23e00640 1 -- 172.21.15.2:0/1506818537 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1400cd80 con 0x7f6b2c068eb0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c19a200 con 0x7f6b2c068eb0 2024-03-11T18:43:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b23e00640 1 -- 172.21.15.2:0/1506818537 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b14018920 con 0x7f6b2c068eb0 2024-03-11T18:43:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.022+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c1a6490 con 0x7f6b2c068eb0 2024-03-11T18:43:36.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.025+0000 7f6b23e00640 1 -- 172.21.15.2:0/1506818537 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f6b14018a80 con 0x7f6b2c068eb0 2024-03-11T18:43:36.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.026+0000 7f6b3522d740 1 Processor -- start 2024-03-11T18:43:36.028 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.026+0000 7f6b3522d740 1 -- start start 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f588620 con 0x56082f5ccff0 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f53a8c0 con 0x56082f5d6570 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b3522d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f537df0 con 0x56082f5cdd60 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b2be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.030 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b2aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b2be00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:51838/0 (socket says 172.21.15.2:51838) 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b2be00640 1 -- 172.21.15.2:0/2070457759 learned_addr learned my addr 172.21.15.2:0/2070457759 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b2b400640 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.027+0000 7f6b23e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 0x7f6b100a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2070457759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 msgr2=0x56082f5ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b23e00640 1 -- 172.21.15.2:0/1506818537 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f6b14014070 con 0x7f6b2c068eb0 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2070457759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 msgr2=0x56082f5cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2070457759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56082f4b4b00 con 0x56082f5d6570 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2be00640 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:36.031 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 0x7f6b100a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d8960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c037040 tx=0x7f6b1c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31fcd44653b62674 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b20c00640 1 -- 172.21.15.2:0/2070457759 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c034440 con 0x56082f5d6570 2024-03-11T18:43:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b20c00640 1 -- 172.21.15.2:0/2070457759 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c034a80 con 0x56082f5d6570 2024-03-11T18:43:36.035 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.028+0000 7f6b20c00640 1 -- 172.21.15.2:0/2070457759 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c003850 con 0x56082f5d6570 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 -- 172.21.15.2:0/2070457759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 msgr2=0x56082f5d8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d8960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c037040 tx=0x7f6b1c0040c0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 0x7f6b100a40c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400f7e0 tx=0x7f6b24011040 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 -- 172.21.15.2:0/2070457759 shutdown_connections 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d8960 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2070457759 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5ccab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 -- 172.21.15.2:0/2070457759 >> 172.21.15.2:0/2070457759 conn(0x56082f5944b0 msgr2=0x56082f5c7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 -- 172.21.15.2:0/2070457759 shutdown_connections 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.029+0000 7f6b3522d740 1 -- 172.21.15.2:0/2070457759 wait complete. 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.030+0000 7f6b3522d740 1 -- start start 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.030+0000 7f6b3522d740 1 Processor -- start 2024-03-11T18:43:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f5335d0 con 0x56082f5ccff0 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f4d8b50 con 0x56082f5d6570 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f4d87f0 con 0x56082f5cdd60 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b3522d740 10 client.0 _ll_register_callbacks cb 0x56082f4ec550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55348/0 (socket says 172.21.15.2:55348) 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 -- 172.21.15.2:0/1126029262 learned_addr learned my addr 172.21.15.2:0/1126029262 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 -- 172.21.15.2:0/1126029262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 msgr2=0x56082f5d3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.037 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 -- 172.21.15.2:0/1126029262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 msgr2=0x56082f5d3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.031+0000 7f6b2be00640 1 -- 172.21.15.2:0/1126029262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c110860 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b2be00640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400dc80 tx=0x7f6b240095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba0b74a52403345e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b19600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24003b70 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56082f505cd0 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f6b2401be40 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b2400dec0 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b3522d740 10 client.26267 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.032+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56082f531c50 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56082f505cd0 con 0x56082f5d6570 2024-03-11T18:43:36.038 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b24026660 con 0x56082f5d6570 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 10 client.26267 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 10 client.26267 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 20 client.26267 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f5e66a0 0x56082f5e8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 4 client.26267 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56082f529ce0 con 0x56082f5e66a0 2024-03-11T18:43:36.039 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.033+0000 7f6b2b400640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f5e66a0 0x56082f5e8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.034+0000 7f6b2b400640 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f5e66a0 0x56082f5e8a90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dcc8d02894b1527 server_cookie=35700c9ac8d7afeb in_seq=0 out_seq=0 2024-03-11T18:43:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.034+0000 7f6b19600640 10 client.26267 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b19600640 1 -- 172.21.15.2:0/1126029262 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56082f529ce0 con 0x56082f5e66a0 2024-03-11T18:43:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b19600640 10 client.26267 handle_command_reply: tid=0 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 client.26267 shutdown 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 msgr2=0x56082f5d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400dc80 tx=0x7f6b240095c0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 shutdown_connections 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f5e66a0 0x56082f5e8a90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5d6570 0x56082f5d4090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b19600640 0 client.26267 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 --2- 172.21.15.2:0/1126029262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 >> 172.21.15.2:0/1126029262 conn(0x56082f5944b0 msgr2=0x56082f5c9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 shutdown_connections 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 1 -- 172.21.15.2:0/1126029262 wait complete. 2024-03-11T18:43:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.035+0000 7f6b3522d740 20 client.26267 trim_cache size 0 max 16384 2024-03-11T18:43:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 Processor -- start 2024-03-11T18:43:36.054 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 -- start start 2024-03-11T18:43:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 0x56082f5dd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f27be60 con 0x56082f5ccff0 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.052+0000 7f6b3522d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f5d6320 con 0x56082f5cdd60 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b3522d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f529ce0 con 0x56082f5d6570 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55356/0 (socket says 172.21.15.2:55356) 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2249982014 learned_addr learned my addr 172.21.15.2:0/2249982014 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2b400640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2be00640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 0x56082f5dd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2249982014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 msgr2=0x56082f5dd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 0x56082f5dd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2249982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 msgr2=0x56082f5d5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2aa00640 1 -- 172.21.15.2:0/2249982014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56082f4ce530 con 0x56082f5cdd60 2024-03-11T18:43:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2b400640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.053+0000 7f6b2be00640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 0x56082f5dd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:43:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.054+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56082f5cda90 tx=0x7f6b1c041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26f80645aee80bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.059+0000 7f6b1be00640 1 -- 172.21.15.2:0/2249982014 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c04d040 con 0x56082f5cdd60 2024-03-11T18:43:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.059+0000 7f6b1be00640 1 -- 172.21.15.2:0/2249982014 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c0044b0 con 0x56082f5cdd60 2024-03-11T18:43:36.061 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.059+0000 7f6b1be00640 1 -- 172.21.15.2:0/2249982014 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0035f0 con 0x56082f5cdd60 2024-03-11T18:43:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 -- 172.21.15.2:0/2249982014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 msgr2=0x56082f5dd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.062 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56082f5cda90 tx=0x7f6b1c041040 comp rx=0 tx=0).stop 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 -- 172.21.15.2:0/2249982014 shutdown_connections 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5d6570 0x56082f5dd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5cdd60 0x56082f5dd330 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 --2- 172.21.15.2:0/2249982014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5ccff0 0x56082f5d5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 -- 172.21.15.2:0/2249982014 >> 172.21.15.2:0/2249982014 conn(0x56082f5944b0 msgr2=0x56082f5d1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 -- 172.21.15.2:0/2249982014 shutdown_connections 2024-03-11T18:43:36.063 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.060+0000 7f6b3522d740 1 -- 172.21.15.2:0/2249982014 wait complete. 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.061+0000 7f6b3522d740 1 -- start start 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.061+0000 7f6b3522d740 1 Processor -- start 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5d6570 0x56082f5d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f5d0ce0 con 0x56082f5d6570 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f5d0e50 con 0x56082f5ccff0 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082f6453a0 con 0x56082f5cdd60 2024-03-11T18:43:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b3522d740 10 client.0 _ll_register_callbacks cb 0x7f6b24009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:55366/0 (socket says 172.21.15.2:55366) 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 -- 172.21.15.2:0/18928438 learned_addr learned my addr 172.21.15.2:0/18928438 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2be00640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5d6570 0x56082f5d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 -- 172.21.15.2:0/18928438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 msgr2=0x56082f5d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 -- 172.21.15.2:0/18928438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5d6570 msgr2=0x56082f5d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5d6570 0x56082f5d05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.062+0000 7f6b2b400640 1 -- 172.21.15.2:0/18928438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56082f531c50 con 0x56082f5ccff0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b2b400640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6b1409a4e0 tx=0x7f6b140d0280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9156bd1767d246 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b19600640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b140b2070 con 0x56082f5ccff0 2024-03-11T18:43:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f6b1409a7e0 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56082f558d30 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b1409e930 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b3522d740 10 client.26273 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56082f4b4b00 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.063+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b140b7030 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b140a6080 con 0x56082f5ccff0 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 10 client.26273 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 10 client.26273 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 20 client.26273 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f646b70 0x56082f648f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 4 client.26273 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56082f233600 con 0x56082f646b70 2024-03-11T18:43:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f646b70 0x56082f648f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b2aa00640 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f646b70 0x56082f648f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764c1faca6382ef3 server_cookie=58380c3e2e9979 in_seq=0 out_seq=0 2024-03-11T18:43:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.064+0000 7f6b19600640 10 client.26273 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:36.305 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:36 smithi002 ceph-mon[31328]: pgmap v309: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-03-11T18:43:36.321 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b19600640 1 -- 172.21.15.2:0/18928438 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56082f233600 con 0x56082f646b70 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b19600640 10 client.26273 handle_command_reply: tid=0 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 client.26273 shutdown 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 msgr2=0x56082f5dd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6b1409a4e0 tx=0x7f6b140d0280 comp rx=0 tx=0).stop 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 shutdown_connections 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x56082f646b70 0x56082f648f90 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56082f5d6570 0x56082f5d05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x56082f5cdd60 0x56082f5d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 --2- 172.21.15.2:0/18928438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56082f5ccff0 0x56082f5dd730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b19600640 0 client.26273 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 >> 172.21.15.2:0/18928438 conn(0x56082f5944b0 msgr2=0x56082f5c64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.322 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 shutdown_connections 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 1 -- 172.21.15.2:0/18928438 wait complete. 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.319+0000 7f6b3522d740 20 client.26273 trim_cache size 0 max 16384 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:43:36.323 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 msgr2=0x7f6b100a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 0x7f6b100a40c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400f7e0 tx=0x7f6b24011040 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 msgr2=0x7f6b2c19f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400e9f0 tx=0x7f6b1400eeb0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 shutdown_connections 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6b100a1c00 0x7f6b100a40c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6b2c10d770 0x7f6b2c1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b2c101f10 0x7f6b2c19f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 --2- 172.21.15.2:0/1506818537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b2c068eb0 0x7f6b2c19f3f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 >> 172.21.15.2:0/1506818537 conn(0x7f6b2c0759e0 msgr2=0x7f6b2c10e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.320+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 shutdown_connections 2024-03-11T18:43:36.324 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:36.321+0000 7f6b30e00640 1 -- 172.21.15.2:0/1506818537 wait complete. 2024-03-11T18:43:36.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:36 smithi165 ceph-mon[36634]: pgmap v309: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-03-11T18:43:36.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:36 smithi023 ceph-mon[36406]: pgmap v309: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2024-03-11T18:43:36.449 INFO:tasks.workunit.client.0.smithi002.stdout: 1 741565 7.33 MB/sec execute 396 sec latency 130.784 ms 2024-03-11T18:43:37.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 742599 7.33 MB/sec execute 397 sec latency 31.786 ms 2024-03-11T18:43:38.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:38 smithi165 ceph-mon[36634]: pgmap v310: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-03-11T18:43:38.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:38 smithi023 ceph-mon[36406]: pgmap v310: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-03-11T18:43:38.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 743654 7.32 MB/sec execute 398 sec latency 31.342 ms 2024-03-11T18:43:38.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:38 smithi002 ceph-mon[31328]: pgmap v310: 129 pgs: 129 active+clean; 104 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-03-11T18:43:39.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 745648 7.33 MB/sec execute 399 sec latency 35.622 ms 2024-03-11T18:43:40.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:40 smithi165 ceph-mon[36634]: pgmap v311: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-03-11T18:43:40.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:40 smithi023 ceph-mon[36406]: pgmap v311: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-03-11T18:43:40.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 746563 7.32 MB/sec execute 400 sec latency 28.217 ms 2024-03-11T18:43:40.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:40 smithi002 ceph-mon[31328]: pgmap v311: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 229 op/s 2024-03-11T18:43:41.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 748054 7.32 MB/sec execute 401 sec latency 25.775 ms 2024-03-11T18:43:42.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:42 smithi165 ceph-mon[36634]: pgmap v312: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.8 MiB/s wr, 235 op/s 2024-03-11T18:43:42.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:42 smithi023 ceph-mon[36406]: pgmap v312: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.8 MiB/s wr, 235 op/s 2024-03-11T18:43:42.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 749700 7.33 MB/sec execute 402 sec latency 12.269 ms 2024-03-11T18:43:42.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:42 smithi002 ceph-mon[31328]: pgmap v312: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.8 MiB/s wr, 235 op/s 2024-03-11T18:43:43.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 750828 7.33 MB/sec execute 403 sec latency 31.892 ms 2024-03-11T18:43:44.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:44 smithi165 ceph-mon[36634]: pgmap v313: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-03-11T18:43:44.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:44 smithi023 ceph-mon[36406]: pgmap v313: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-03-11T18:43:44.450 INFO:tasks.workunit.client.0.smithi002.stdout: 1 752521 7.33 MB/sec execute 404 sec latency 65.332 ms 2024-03-11T18:43:44.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:44 smithi002 ceph-mon[31328]: pgmap v313: 129 pgs: 129 active+clean; 101 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.3 MiB/s wr, 210 op/s 2024-03-11T18:43:45.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 753500 7.32 MB/sec execute 405 sec latency 28.605 ms 2024-03-11T18:43:46.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:46 smithi165 ceph-mon[36634]: pgmap v314: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 270 op/s 2024-03-11T18:43:46.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:46 smithi023 ceph-mon[36406]: pgmap v314: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 270 op/s 2024-03-11T18:43:46.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 754661 7.32 MB/sec execute 406 sec latency 23.183 ms 2024-03-11T18:43:46.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:46 smithi002 ceph-mon[31328]: pgmap v314: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 270 op/s 2024-03-11T18:43:47.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 756566 7.33 MB/sec execute 407 sec latency 37.333 ms 2024-03-11T18:43:48.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:48 smithi165 ceph-mon[36634]: pgmap v315: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-03-11T18:43:48.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:48 smithi023 ceph-mon[36406]: pgmap v315: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-03-11T18:43:48.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 757296 7.32 MB/sec execute 408 sec latency 31.459 ms 2024-03-11T18:43:48.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:48 smithi002 ceph-mon[31328]: pgmap v315: 129 pgs: 129 active+clean; 107 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-03-11T18:43:49.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 758696 7.32 MB/sec execute 409 sec latency 29.416 ms 2024-03-11T18:43:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 -- 172.21.15.2:0/1781584123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf701021b0 msgr2=0x7fbf70102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 --2- 172.21.15.2:0/1781584123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf701021b0 0x7fbf70102590 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbf60009900 tx=0x7fbf6002bcc0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 -- 172.21.15.2:0/1781584123 shutdown_connections 2024-03-11T18:43:49.475 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 --2- 172.21.15.2:0/1781584123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf7010b580 0x7fbf7010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 --2- 172.21.15.2:0/1781584123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf70102ad0 0x7fbf7010afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 --2- 172.21.15.2:0/1781584123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf701021b0 0x7fbf70102590 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 -- 172.21.15.2:0/1781584123 >> 172.21.15.2:0/1781584123 conn(0x7fbf700fb970 msgr2=0x7fbf700fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 -- 172.21.15.2:0/1781584123 shutdown_connections 2024-03-11T18:43:49.476 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.472+0000 7fbf6f200640 1 -- 172.21.15.2:0/1781584123 wait complete. 2024-03-11T18:43:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.473+0000 7fbf6f200640 1 Processor -- start 2024-03-11T18:43:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.473+0000 7fbf6f200640 1 -- start start 2024-03-11T18:43:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf70102ad0 0x7fbf7019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 0x7fbf70199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70199c60 con 0x7fbf7010b580 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70199dd0 con 0x7fbf701021b0 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6f200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf70199f90 con 0x7fbf70102ad0 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54346/0 (socket says 172.21.15.2:54346) 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 -- 172.21.15.2:0/81795823 learned_addr learned my addr 172.21.15.2:0/81795823 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf67e00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf70102ad0 0x7fbf7019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6e800640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 0x7fbf70199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 -- 172.21.15.2:0/81795823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf70102ad0 msgr2=0x7fbf7019f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf70102ad0 0x7fbf7019f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 -- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 msgr2=0x7fbf70199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 0x7fbf70199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6de00640 1 -- 172.21.15.2:0/81795823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60009560 con 0x7fbf701021b0 2024-03-11T18:43:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.474+0000 7fbf6e800640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 0x7fbf70199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:49.479 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf6de00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf60000c00 tx=0x7fbf6002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf66a00640 1 -- 172.21.15.2:0/81795823 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6002dbd0 con 0x7fbf701021b0 2024-03-11T18:43:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf66a00640 1 -- 172.21.15.2:0/81795823 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf6002dd30 con 0x7fbf701021b0 2024-03-11T18:43:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf66a00640 1 -- 172.21.15.2:0/81795823 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60003b20 con 0x7fbf701021b0 2024-03-11T18:43:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7019a210 con 0x7fbf701021b0 2024-03-11T18:43:49.480 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.475+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf701a6500 con 0x7fbf701021b0 2024-03-11T18:43:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.478+0000 7fbf66a00640 1 -- 172.21.15.2:0/81795823 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fbf60003c80 con 0x7fbf701021b0 2024-03-11T18:43:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.478+0000 7fbf777c1740 1 Processor -- start 2024-03-11T18:43:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 -- start start 2024-03-11T18:43:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad922d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad9238960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad91e8620 con 0x557ad922cff0 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad919a8c0 con 0x557ad922dd60 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf777c1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad9197df0 con 0x557ad9236570 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54350/0 (socket says 172.21.15.2:54350) 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf6e800640 1 -- 172.21.15.2:0/1774957768 learned_addr learned my addr 172.21.15.2:0/1774957768 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf6de00640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad922d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.479+0000 7fbf67e00640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad9238960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 -- 172.21.15.2:0/1774957768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 msgr2=0x557ad9238960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad9238960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 -- 172.21.15.2:0/1774957768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 msgr2=0x557ad922d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad922d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 -- 172.21.15.2:0/1774957768 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ad9114b00 con 0x557ad922dd60 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6de00640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad922d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009fd0 tx=0x7fbf68010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a812a1089dcc9f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf5f000640 1 -- 172.21.15.2:0/1774957768 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68011700 con 0x557ad922dd60 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf66a00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 0x7fbf540a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf5f000640 1 -- 172.21.15.2:0/1774957768 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf68015070 con 0x557ad922dd60 2024-03-11T18:43:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.480+0000 7fbf5f000640 1 -- 172.21.15.2:0/1774957768 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6801bde0 con 0x557ad922dd60 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf66a00640 1 -- 172.21.15.2:0/81795823 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fbf600f0100 con 0x7fbf701021b0 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf6de00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 0x7fbf540a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 -- 172.21.15.2:0/1774957768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 msgr2=0x557ad922cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf68009fd0 tx=0x7fbf68010860 comp rx=0 tx=0).stop 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 -- 172.21.15.2:0/1774957768 shutdown_connections 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad9238960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad922d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1774957768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad922cab0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 -- 172.21.15.2:0/1774957768 >> 172.21.15.2:0/1774957768 conn(0x557ad91f44b0 msgr2=0x557ad9227ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.481+0000 7fbf777c1740 1 -- 172.21.15.2:0/1774957768 shutdown_connections 2024-03-11T18:43:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.482+0000 7fbf6de00640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 0x7fbf540a40c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x557ad922e4c0 tx=0x7fbf6002bcc0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.482+0000 7fbf777c1740 1 -- 172.21.15.2:0/1774957768 wait complete. 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.482+0000 7fbf777c1740 1 -- start start 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.482+0000 7fbf777c1740 1 Processor -- start 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922cff0 0x557ad92335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad9236570 0x557ad9234040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad91935d0 con 0x557ad9236570 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad9138b50 con 0x557ad922dd60 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad91387f0 con 0x557ad922cff0 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf777c1740 10 client.0 _ll_register_callbacks cb 0x557ad914c550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54366/0 (socket says 172.21.15.2:54366) 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 -- 172.21.15.2:0/1419500435 learned_addr learned my addr 172.21.15.2:0/1419500435 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922cff0 0x557ad92335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad9236570 0x557ad9234040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 -- 172.21.15.2:0/1419500435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922cff0 msgr2=0x557ad92335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922cff0 0x557ad92335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.483+0000 7fbf67e00640 1 -- 172.21.15.2:0/1419500435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad9236570 msgr2=0x557ad9234040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf67e00640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad9236570 0x557ad9234040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf67e00640 1 -- 172.21.15.2:0/1419500435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68009c30 con 0x557ad922dd60 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf67e00640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf58012700 tx=0x7fbf58012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3c27951ba81314 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf53400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf58017070 con 0x557ad922dd60 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ad9165cd0 con 0x557ad922dd60 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fbf58004050 con 0x557ad922dd60 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf58004210 con 0x557ad922dd60 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf777c1740 10 client.26285 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.484+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557ad9191c50 con 0x557ad922dd60 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557ad9165cd0 con 0x557ad922dd60 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf580039d0 con 0x557ad922dd60 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 10 client.26285 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 10 client.26285 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 20 client.26285 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad9246660 0x557ad9248a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 4 client.26285 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ad9189ce0 con 0x557ad9246660 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.485+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad9246660 0x557ad9248a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.486+0000 7fbf53400640 10 client.26285 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.486+0000 7fbf6e800640 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad9246660 0x557ad9248a80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caed8ff4b9591ad1 server_cookie=d1a2113287a9379e in_seq=0 out_seq=0 2024-03-11T18:43:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.486+0000 7fbf53400640 1 -- 172.21.15.2:0/1419500435 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557ad9189ce0 con 0x557ad9246660 2024-03-11T18:43:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.486+0000 7fbf53400640 10 client.26285 handle_command_reply: tid=0 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.486+0000 7fbf777c1740 1 client.26285 shutdown 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 msgr2=0x557ad9233b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbf58012700 tx=0x7fbf58012bc0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 shutdown_connections 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad9246660 0x557ad9248a80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad9236570 0x557ad9234040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922dd60 0x557ad9233b00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 --2- 172.21.15.2:0/1419500435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922cff0 0x557ad92335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf53400640 0 client.26285 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 >> 172.21.15.2:0/1419500435 conn(0x557ad91f44b0 msgr2=0x557ad9229a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 shutdown_connections 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 1 -- 172.21.15.2:0/1419500435 wait complete. 2024-03-11T18:43:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.487+0000 7fbf777c1740 20 client.26285 trim_cache size 0 max 16384 2024-03-11T18:43:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.503+0000 7fbf777c1740 1 Processor -- start 2024-03-11T18:43:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.503+0000 7fbf777c1740 1 -- start start 2024-03-11T18:43:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.503+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad9235cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.507 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.503+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 0x557ad9231e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 0x557ad92323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf777c1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad8edbe60 con 0x557ad922cff0 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf777c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad9244e80 con 0x557ad9236570 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf777c1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad9228ae0 con 0x557ad922dd60 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf67e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 0x557ad9231e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf67e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 0x557ad9231e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:42832/0 (socket says 172.21.15.2:42832) 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf67e00640 1 -- 172.21.15.2:0/451154086 learned_addr learned my addr 172.21.15.2:0/451154086 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad9235cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 0x557ad92323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 -- 172.21.15.2:0/451154086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 msgr2=0x557ad9231e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 0x557ad9231e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 -- 172.21.15.2:0/451154086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 msgr2=0x557ad9235cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.508 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad9235cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.504+0000 7fbf6e800640 1 -- 172.21.15.2:0/451154086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ad912e530 con 0x557ad9236570 2024-03-11T18:43:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.505+0000 7fbf6e800640 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 0x557ad92323d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbf6801ba40 tx=0x7fbf68010fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e4923726e49a061 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.505+0000 7fbf5e600640 1 -- 172.21.15.2:0/451154086 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6801dd40 con 0x557ad9236570 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.505+0000 7fbf5e600640 1 -- 172.21.15.2:0/451154086 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf68030070 con 0x557ad9236570 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.505+0000 7fbf5e600640 1 -- 172.21.15.2:0/451154086 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf68029620 con 0x557ad9236570 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- 172.21.15.2:0/451154086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 msgr2=0x557ad92323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 0x557ad92323d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbf6801ba40 tx=0x7fbf68010fd0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- 172.21.15.2:0/451154086 shutdown_connections 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad9236570 0x557ad92323d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad922dd60 0x557ad9231e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 --2- 172.21.15.2:0/451154086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922cff0 0x557ad9235cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- 172.21.15.2:0/451154086 >> 172.21.15.2:0/451154086 conn(0x557ad91f44b0 msgr2=0x557ad9227810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- 172.21.15.2:0/451154086 shutdown_connections 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- 172.21.15.2:0/451154086 wait complete. 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.506+0000 7fbf777c1740 1 -- start start 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 Processor -- start 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922dd60 0x557ad923bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad923c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad923cac0 con 0x557ad922dd60 2024-03-11T18:43:49.513 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad929cb70 con 0x557ad922cff0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ad929ccb0 con 0x557ad9236570 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf777c1740 10 client.0 _ll_register_callbacks cb 0x7fbf580097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf6e800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:54398/0 (socket says 172.21.15.2:54398) 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf6e800640 1 -- 172.21.15.2:0/2052097818 learned_addr learned my addr 172.21.15.2:0/2052097818 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.507+0000 7fbf6e800640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad923c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf67e00640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922dd60 0x557ad923bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 -- 172.21.15.2:0/2052097818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 msgr2=0x557ad923c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad923c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 -- 172.21.15.2:0/2052097818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922dd60 msgr2=0x557ad923bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922dd60 0x557ad923bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 -- 172.21.15.2:0/2052097818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ad9114b00 con 0x557ad922cff0 2024-03-11T18:43:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf6e800640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbf68004e60 tx=0x7fbf68013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0ef02c273841cff server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf53400640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6801db40 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mon.2 v2:172.21.15.23:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fbf68017070 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.508+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ad91b8d30 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf6801dca0 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 10 client.26291 fetch_fsmap learned FSMap version 31 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557ad91542c0 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mon.2 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf6804a080 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mon.2 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf6802ad20 con 0x557ad922cff0 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 10 client.26291 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 10 client.26291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 20 client.26291 populate_metadata read hostname 'smithi002' 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad929e0e0 0x557ad92a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 4 client.26291 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:43:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557ad8e93600 con 0x557ad929e0e0 2024-03-11T18:43:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.509+0000 7fbf67e00640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad929e0e0 0x557ad92a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:43:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.510+0000 7fbf67e00640 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad929e0e0 0x557ad92a04d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83443a92dbb43809 server_cookie=1304bfbf6d7ec104 in_seq=0 out_seq=0 2024-03-11T18:43:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.510+0000 7fbf53400640 10 client.26291 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.749+0000 7fbf53400640 1 -- 172.21.15.2:0/2052097818 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557ad8e93600 con 0x557ad929e0e0 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.749+0000 7fbf53400640 10 client.26291 handle_command_reply: tid=0 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.749+0000 7fbf777c1740 1 client.26291 shutdown 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 msgr2=0x557ad923d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbf68004e60 tx=0x7fbf68013040 comp rx=0 tx=0).stop 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 shutdown_connections 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x557ad929e0e0 0x557ad92a04d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x557ad9236570 0x557ad923c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x557ad922dd60 0x557ad923bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 --2- 172.21.15.2:0/2052097818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557ad922cff0 0x557ad923d910 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf53400640 0 client.26291 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 >> 172.21.15.2:0/2052097818 conn(0x557ad91f44b0 msgr2=0x557ad9227810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.750+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 shutdown_connections 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.751+0000 7fbf777c1740 1 -- 172.21.15.2:0/2052097818 wait complete. 2024-03-11T18:43:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.751+0000 7fbf777c1740 20 client.26291 trim_cache size 0 max 16384 2024-03-11T18:43:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:43:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:43:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:43:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 msgr2=0x7fbf540a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 0x7fbf540a40c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x557ad922e4c0 tx=0x7fbf6002bcc0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 msgr2=0x7fbf7019f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf60000c00 tx=0x7fbf6002c6c0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 shutdown_connections 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fbf540a1c00 0x7fbf540a40c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf7010b580 0x7fbf70199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbf70102ad0 0x7fbf7019f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 --2- 172.21.15.2:0/81795823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf701021b0 0x7fbf7019f460 secure :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf60000c00 tx=0x7fbf6002c6c0 comp rx=0 tx=0).stop 2024-03-11T18:43:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.753+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 >> 172.21.15.2:0/81795823 conn(0x7fbf700fb970 msgr2=0x7fbf7010c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:43:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.754+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 shutdown_connections 2024-03-11T18:43:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:43:49.754+0000 7fbf6f200640 1 -- 172.21.15.2:0/81795823 wait complete. 2024-03-11T18:43:50.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:50 smithi165 ceph-mon[36634]: pgmap v316: 129 pgs: 129 active+clean; 111 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-03-11T18:43:50.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:50 smithi023 ceph-mon[36406]: pgmap v316: 129 pgs: 129 active+clean; 111 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-03-11T18:43:50.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 760108 7.33 MB/sec execute 410 sec latency 103.620 ms 2024-03-11T18:43:50.502 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:50 smithi002 ceph-mon[31328]: pgmap v316: 129 pgs: 129 active+clean; 111 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 261 op/s 2024-03-11T18:43:51.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 760935 7.31 MB/sec execute 411 sec latency 25.380 ms 2024-03-11T18:43:52.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:52 smithi165 ceph-mon[36634]: pgmap v317: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:43:52.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:52 smithi023 ceph-mon[36406]: pgmap v317: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:43:52.451 INFO:tasks.workunit.client.0.smithi002.stdout: 1 762344 7.31 MB/sec execute 412 sec latency 23.213 ms 2024-03-11T18:43:52.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:52 smithi002 ceph-mon[31328]: pgmap v317: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-03-11T18:43:53.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 763836 7.32 MB/sec execute 413 sec latency 14.249 ms 2024-03-11T18:43:54.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:54 smithi165 ceph-mon[36634]: pgmap v318: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 134 op/s 2024-03-11T18:43:54.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:54 smithi023 ceph-mon[36406]: pgmap v318: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 134 op/s 2024-03-11T18:43:54.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 764660 7.31 MB/sec execute 414 sec latency 70.190 ms 2024-03-11T18:43:54.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:54 smithi002 ceph-mon[31328]: pgmap v318: 129 pgs: 129 active+clean; 110 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 134 op/s 2024-03-11T18:43:55.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 766168 7.31 MB/sec execute 415 sec latency 32.090 ms 2024-03-11T18:43:56.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:56 smithi165 ceph-mon[36634]: pgmap v319: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2024-03-11T18:43:56.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:56 smithi023 ceph-mon[36406]: pgmap v319: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2024-03-11T18:43:56.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 767431 7.31 MB/sec execute 416 sec latency 14.147 ms 2024-03-11T18:43:56.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:56 smithi002 ceph-mon[31328]: pgmap v319: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 270 op/s 2024-03-11T18:43:57.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 768367 7.31 MB/sec execute 417 sec latency 48.793 ms 2024-03-11T18:43:58.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:43:58 smithi165 ceph-mon[36634]: pgmap v320: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.8 MiB/s wr, 210 op/s 2024-03-11T18:43:58.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:43:58 smithi023 ceph-mon[36406]: pgmap v320: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.8 MiB/s wr, 210 op/s 2024-03-11T18:43:58.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 769851 7.31 MB/sec execute 418 sec latency 27.888 ms 2024-03-11T18:43:58.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:43:58 smithi002 ceph-mon[31328]: pgmap v320: 129 pgs: 129 active+clean; 102 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.8 MiB/s wr, 210 op/s 2024-03-11T18:43:59.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 770971 7.31 MB/sec execute 419 sec latency 36.771 ms 2024-03-11T18:44:00.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:00 smithi165 ceph-mon[36634]: pgmap v321: 129 pgs: 129 active+clean; 105 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-11T18:44:00.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:00 smithi023 ceph-mon[36406]: pgmap v321: 129 pgs: 129 active+clean; 105 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-11T18:44:00.452 INFO:tasks.workunit.client.0.smithi002.stdout: 1 771995 7.30 MB/sec execute 420 sec latency 49.817 ms 2024-03-11T18:44:00.705 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:00 smithi002 ceph-mon[31328]: pgmap v321: 129 pgs: 129 active+clean; 105 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-11T18:44:01.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 772478 7.29 MB/sec execute 421 sec latency 576.578 ms 2024-03-11T18:44:02.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:02 smithi023 ceph-mon[36406]: pgmap v322: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-03-11T18:44:02.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:02 smithi165 ceph-mon[36634]: pgmap v322: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-03-11T18:44:02.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 772770 7.27 MB/sec execute 422 sec latency 988.072 ms 2024-03-11T18:44:02.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:02 smithi002 ceph-mon[31328]: pgmap v322: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 1.9 MiB/s wr, 219 op/s 2024-03-11T18:44:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 -- 172.21.15.2:0/3277533592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab80fff70 msgr2=0x7f1ab8100350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 --2- 172.21.15.2:0/3277533592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab80fff70 0x7f1ab8100350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009860 tx=0x7f1aac02bb70 comp rx=0 tx=0).stop 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 -- 172.21.15.2:0/3277533592 shutdown_connections 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 --2- 172.21.15.2:0/3277533592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab810db20 0x7f1ab810ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 --2- 172.21.15.2:0/3277533592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab8100890 0x7f1ab810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 --2- 172.21.15.2:0/3277533592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab80fff70 0x7f1ab8100350 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 -- 172.21.15.2:0/3277533592 >> 172.21.15.2:0/3277533592 conn(0x7f1ab8077190 msgr2=0x7f1ab80775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 -- 172.21.15.2:0/3277533592 shutdown_connections 2024-03-11T18:44:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.935+0000 7f1abf800640 1 -- 172.21.15.2:0/3277533592 wait complete. 2024-03-11T18:44:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.936+0000 7f1abf800640 1 Processor -- start 2024-03-11T18:44:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 -- start start 2024-03-11T18:44:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 0x7f1ab819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 0x7f1ab819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab810db20 0x7f1ab8199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8199fc0 con 0x7f1ab810db20 2024-03-11T18:44:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab819a130 con 0x7f1ab8100890 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abf800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab819a2f0 con 0x7f1ab80fff70 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.937+0000 7f1abce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 0x7f1ab819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1abce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 0x7f1ab819f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34444/0 (socket says 172.21.15.2:34444) 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1abce00640 1 -- 172.21.15.2:0/3122709121 learned_addr learned my addr 172.21.15.2:0/3122709121 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 0x7f1ab819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1abd800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab810db20 0x7f1ab8199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3122709121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 msgr2=0x7f1ab819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 0x7f1ab819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab810db20 msgr2=0x7f1ab8199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab810db20 0x7f1ab8199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3122709121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac009490 con 0x7f1ab8100890 2024-03-11T18:44:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.938+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 0x7f1ab819fcb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400e9f0 tx=0x7f1aa400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.939+0000 7f1ab5600640 1 -- 172.21.15.2:0/3122709121 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa400ccc0 con 0x7f1ab8100890 2024-03-11T18:44:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.939+0000 7f1ab5600640 1 -- 172.21.15.2:0/3122709121 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aa400ce20 con 0x7f1ab8100890 2024-03-11T18:44:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.939+0000 7f1ab5600640 1 -- 172.21.15.2:0/3122709121 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4018920 con 0x7f1ab8100890 2024-03-11T18:44:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.939+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab819a5d0 con 0x7f1ab8100890 2024-03-11T18:44:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.939+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab81a6810 con 0x7f1ab8100890 2024-03-11T18:44:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.942+0000 7f1ab5600640 1 -- 172.21.15.2:0/3122709121 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f1aa4018a80 con 0x7f1ab8100890 2024-03-11T18:44:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.943+0000 7f1ac2444740 1 Processor -- start 2024-03-11T18:44:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.943+0000 7f1ac2444740 1 -- start start 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a13b20 0x559003a12870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a1e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590039ce620 con 0x559003a1c330 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590039808c0 con 0x559003a13b20 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ac2444740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900397ddf0 con 0x559003a12db0 2024-03-11T18:44:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ab5600640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 0x7f1aa00a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1abce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1abce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34456/0 (socket says 172.21.15.2:34456) 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1abce00640 1 -- 172.21.15.2:0/3891990526 learned_addr learned my addr 172.21.15.2:0/3891990526 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1abd800640 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a13b20 0x559003a12870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1abd800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 0x7f1aa00a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.944+0000 7f1ab5600640 1 -- 172.21.15.2:0/3122709121 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f1aa4014070 con 0x7f1ab8100890 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a1e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 -- 172.21.15.2:0/3891990526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a13b20 msgr2=0x559003a12870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a13b20 0x559003a12870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 -- 172.21.15.2:0/3891990526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 msgr2=0x559003a1e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a1e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 -- 172.21.15.2:0/3891990526 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590038fab00 con 0x559003a12db0 2024-03-11T18:44:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1abce00640 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0057c0 tx=0x7f1aac032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f54c212adcfe652e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.945+0000 7f1aaa000640 1 -- 172.21.15.2:0/3891990526 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac004040 con 0x559003a12db0 2024-03-11T18:44:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.946+0000 7f1abd800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 0x7f1aa00a40d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009fd0 tx=0x7f1ab0005bd0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.946+0000 7f1aaa000640 1 -- 172.21.15.2:0/3891990526 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aac034440 con 0x559003a12db0 2024-03-11T18:44:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.946+0000 7f1aaa000640 1 -- 172.21.15.2:0/3891990526 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac02c8d0 con 0x559003a12db0 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.946+0000 7f1ac2444740 1 -- 172.21.15.2:0/3891990526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 msgr2=0x559003a13230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.946+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0057c0 tx=0x7f1aac032450 comp rx=0 tx=0).stop 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 -- 172.21.15.2:0/3891990526 shutdown_connections 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a1e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a12db0 0x559003a13230 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3891990526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a13b20 0x559003a12870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 -- 172.21.15.2:0/3891990526 >> 172.21.15.2:0/3891990526 conn(0x5590039da4b0 msgr2=0x559003a0daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 -- 172.21.15.2:0/3891990526 shutdown_connections 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 -- 172.21.15.2:0/3891990526 wait complete. 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.947+0000 7f1ac2444740 1 -- start start 2024-03-11T18:44:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 Processor -- start 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a19350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a19dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590039795d0 con 0x559003a1c330 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900391eb50 con 0x559003a12db0 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900391e7f0 con 0x559003a13b20 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ac2444740 10 client.0 _ll_register_callbacks cb 0x559003932550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34464/0 (socket says 172.21.15.2:34464) 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1abce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a19350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.948+0000 7f1ab7e00640 1 -- 172.21.15.2:0/2386779641 learned_addr learned my addr 172.21.15.2:0/2386779641 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1abd800640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a19dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.952 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 -- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 msgr2=0x559003a19350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a19350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 -- 172.21.15.2:0/2386779641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 msgr2=0x559003a19dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a19dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 -- 172.21.15.2:0/2386779641 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab8110aa0 con 0x559003a13b20 2024-03-11T18:44:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4096b60 tx=0x7f1aa409a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=625ff27b88d2f75a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1a9ea00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:44:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40cb5f0 con 0x559003a13b20 2024-03-11T18:44:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.949+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55900394bcd0 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f1aa40a2df0 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aa40850b0 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 10 client.26299 fetch_fsmap learned FSMap version 31 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559003977c50 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55900394bcd0 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1aa4099b50 con 0x559003a13b20 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 10 client.26299 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 10 client.26299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 20 client.26299 populate_metadata read hostname 'smithi002' 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.950+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a2c120 0x559003a2e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.951+0000 7f1ac2444740 4 client.26299 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.951+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55900396fce0 con 0x559003a2c120 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.951+0000 7f1abce00640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a2c120 0x559003a2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.951+0000 7f1abce00640 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a2c120 0x559003a2e560 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=309c3644fcedb41b server_cookie=307a6407a49175b1 in_seq=0 out_seq=0 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.951+0000 7f1a9ea00640 10 client.26299 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1a9ea00640 1 -- 172.21.15.2:0/2386779641 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55900396fce0 con 0x559003a2c120 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1a9ea00640 10 client.26299 handle_command_reply: tid=0 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 client.26299 shutdown 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 msgr2=0x559003a19890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4096b60 tx=0x7f1aa409a330 comp rx=0 tx=0).stop 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 shutdown_connections 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a2c120 0x559003a2e560 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a19dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19890 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1a9ea00640 0 client.26299 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2386779641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a19350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.952+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 >> 172.21.15.2:0/2386779641 conn(0x5590039da4b0 msgr2=0x559003a0f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.953+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 shutdown_connections 2024-03-11T18:44:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.953+0000 7f1ac2444740 1 -- 172.21.15.2:0/2386779641 wait complete. 2024-03-11T18:44:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.953+0000 7f1ac2444740 20 client.26299 trim_cache size 0 max 16384 2024-03-11T18:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 Processor -- start 2024-03-11T18:44:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 -- start start 2024-03-11T18:44:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a22fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590036c1e60 con 0x559003a1c330 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900394bcd0 con 0x559003a12db0 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1ac2444740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559003a142b0 con 0x559003a13b20 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.974+0000 7f1abce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:60744/0 (socket says 172.21.15.2:60744) 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 -- 172.21.15.2:0/2389545444 learned_addr learned my addr 172.21.15.2:0/2389545444 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abd800640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a22fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 -- 172.21.15.2:0/2389545444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 msgr2=0x559003a19bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 -- 172.21.15.2:0/2389545444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 msgr2=0x559003a22fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.978 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a22fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 -- 172.21.15.2:0/2389545444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559003914530 con 0x559003a12db0 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abd800640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a22fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1abce00640 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1aac005e00 tx=0x7f1aac02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88ac8f05cf4f819 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.975+0000 7f1aa9600640 1 -- 172.21.15.2:0/2389545444 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac055430 con 0x559003a12db0 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.976+0000 7f1aa9600640 1 -- 172.21.15.2:0/2389545444 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1aac005040 con 0x559003a12db0 2024-03-11T18:44:02.979 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.976+0000 7f1aa9600640 1 -- 172.21.15.2:0/2389545444 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac02c510 con 0x559003a12db0 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.976+0000 7f1ac2444740 1 -- 172.21.15.2:0/2389545444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 msgr2=0x559003a1ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.976+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1aac005e00 tx=0x7f1aac02bb70 comp rx=0 tx=0).stop 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 -- 172.21.15.2:0/2389545444 shutdown_connections 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a1c330 0x559003a22fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a19bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 --2- 172.21.15.2:0/2389545444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a12db0 0x559003a1ba80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 -- 172.21.15.2:0/2389545444 >> 172.21.15.2:0/2389545444 conn(0x5590039da4b0 msgr2=0x559003a17c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 -- 172.21.15.2:0/2389545444 shutdown_connections 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 -- 172.21.15.2:0/2389545444 wait complete. 2024-03-11T18:44:02.982 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 -- start start 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.977+0000 7f1ac2444740 1 Processor -- start 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a12db0 0x559003a232d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a1c330 0x559003a161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x559003a16840 con 0x559003a12db0 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559003a169b0 con 0x559003a1c330 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x559003a16b70 con 0x559003a13b20 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ac2444740 10 client.0 _ll_register_callbacks cb 0x7f1aa400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1abd800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a1c330 0x559003a161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ab7e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:34488/0 (socket says 172.21.15.2:34488) 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3917131456 learned_addr learned my addr 172.21.15.2:0/3917131456 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.978+0000 7f1abce00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a12db0 0x559003a232d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a1c330 msgr2=0x559003a161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.983 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a1c330 0x559003a161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3917131456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a12db0 msgr2=0x559003a232d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a12db0 0x559003a232d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 -- 172.21.15.2:0/3917131456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590038fab00 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ab7e00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1aa409a690 tx=0x7f1aa4010f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb80e36327a072c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1a9ea00640 10 client.0 ms_handle_connect on v2:172.21.15.165:3300/0 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa409a4d0 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mon.1 v2:172.21.15.165:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f1aa40a2df0 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.979+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55900399ed30 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aa40a2960 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 10 client.26305 fetch_fsmap learned FSMap version 31 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55900393a2c0 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mon.1 v2:172.21.15.165:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55900399ed30 con 0x559003a13b20 2024-03-11T18:44:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mon.1 v2:172.21.15.165:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1aa4099750 con 0x559003a13b20 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 10 client.26305 fetch_fsmap finished waiting for FSMap version 31 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 20 client.26305 populate_metadata read hostname 'smithi002' 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a83ac0 0x559003a85ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 4 client.26305 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 --> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559003679600 con 0x559003a83ac0 2024-03-11T18:44:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.980+0000 7f1abce00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a83ac0 0x559003a85ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.981+0000 7f1abce00640 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a83ac0 0x559003a85ee0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb8c04c67cee5628 server_cookie=29a95cabb5613a1b in_seq=0 out_seq=0 2024-03-11T18:44:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:02.981+0000 7f1a9ea00640 10 client.26305 ms_handle_connect on v2:172.21.15.23:6838/2250151518 2024-03-11T18:44:03.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 772787 7.25 MB/sec execute 423 sec latency 161.231 ms 2024-03-11T18:44:04.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.296+0000 7f1a9ea00640 1 -- 172.21.15.2:0/3917131456 <== mds.0 v2:172.21.15.23:6838/2250151518 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559003679600 con 0x559003a83ac0 2024-03-11T18:44:04.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.296+0000 7f1a9ea00640 10 client.26305 handle_command_reply: tid=0 2024-03-11T18:44:04.299 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.296+0000 7f1ac2444740 1 client.26305 shutdown 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 msgr2=0x559003a15ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1aa409a690 tx=0x7f1aa4010f00 comp rx=0 tx=0).stop 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 shutdown_connections 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:6838/2250151518,v1:172.21.15.23:6839/2250151518] conn(0x559003a83ac0 0x559003a85ee0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559003a1c330 0x559003a161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x559003a13b20 0x559003a15ca0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 --2- 172.21.15.2:0/3917131456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x559003a12db0 0x559003a232d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 >> 172.21.15.2:0/3917131456 conn(0x5590039da4b0 msgr2=0x559003a0cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1a9ea00640 0 client.26305 ms_handle_reset on v2:172.21.15.23:6838/2250151518 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 shutdown_connections 2024-03-11T18:44:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 1 -- 172.21.15.2:0/3917131456 wait complete. 2024-03-11T18:44:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.297+0000 7f1ac2444740 20 client.26305 trim_cache size 0 max 16384 2024-03-11T18:44:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:{ 2024-03-11T18:44:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "message": "", 2024-03-11T18:44:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout: "return_code": 0 2024-03-11T18:44:04.301 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stdout:} 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 msgr2=0x7f1aa00a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 0x7f1aa00a40d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009fd0 tx=0x7f1ab0005bd0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 msgr2=0x7f1ab819fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 0x7f1ab819fcb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400e9f0 tx=0x7f1aa400eeb0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 shutdown_connections 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f1aa00a1c10 0x7f1aa00a40d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ab810db20 0x7f1ab8199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ab8100890 0x7f1ab819fcb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 --2- 172.21.15.2:0/3122709121 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab80fff70 0x7f1ab819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:04.302 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.298+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 >> 172.21.15.2:0/3122709121 conn(0x7f1ab8077190 msgr2=0x7f1ab810f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.299+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 shutdown_connections 2024-03-11T18:44:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi002.front.sepia.ceph.com.smithi002.stderr:2024-03-11T18:44:04.299+0000 7f1abf800640 1 -- 172.21.15.2:0/3122709121 wait complete. 2024-03-11T18:44:04.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:04 smithi165 ceph-mon[36634]: pgmap v323: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-03-11T18:44:04.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:04 smithi023 ceph-mon[36406]: pgmap v323: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-03-11T18:44:04.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 773108 7.24 MB/sec execute 424 sec latency 173.947 ms 2024-03-11T18:44:04.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:04 smithi002 ceph-mon[31328]: pgmap v323: 129 pgs: 129 active+clean; 106 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.7 MiB/s wr, 201 op/s 2024-03-11T18:44:05.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 774471 7.25 MB/sec execute 425 sec latency 25.277 ms 2024-03-11T18:44:06.407 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:06 smithi165 ceph-mon[36634]: pgmap v324: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-11T18:44:06.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:06 smithi023 ceph-mon[36406]: pgmap v324: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-11T18:44:06.453 INFO:tasks.workunit.client.0.smithi002.stdout: 1 775473 7.24 MB/sec execute 426 sec latency 24.130 ms 2024-03-11T18:44:06.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:06 smithi002 ceph-mon[31328]: pgmap v324: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2024-03-11T18:44:06.606 INFO:tasks.workunit.client.0.smithi002.stdout:[775573] write failed on handle 12126 (Resource temporarily unavailable) 2024-03-11T18:44:06.607 INFO:tasks.workunit.client.0.smithi002.stdout:Child failed with status 1 2024-03-11T18:44:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:44:06.611 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-11T18:44:06.611 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T18:44:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:08 smithi165 ceph-mon[36634]: pgmap v325: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-11T18:44:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:08 smithi165 ceph-mon[36634]: osdmap e107: 12 total, 12 up, 12 in 2024-03-11T18:44:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:08 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:08 smithi023 ceph-mon[36406]: pgmap v325: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:08 smithi023 ceph-mon[36406]: osdmap e107: 12 total, 12 up, 12 in 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:08 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:08 smithi002 ceph-mon[31328]: pgmap v325: 129 pgs: 129 active+clean; 106 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.5 MiB/s wr, 152 op/s 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:08 smithi002 ceph-mon[31328]: osdmap e107: 12 total, 12 up, 12 in 2024-03-11T18:44:08.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:08 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T18:44:08.413 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-11T18:44:08.634 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1b4633bc9037418db2cde4842135fe40 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_b6a4997be2b8bb74411268cc42753f29ae9a78ad/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-11T18:44:08.636 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-11T18:44:08.646 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:44:08.671 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-11T18:44:08.694 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-11T18:44:08.694 INFO:teuthology.task.exec:Executing custom commands... 2024-03-11T18:44:08.695 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:44:08.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-11T18:44:08.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-11T18:44:09.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-11T18:44:09.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.256 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:09 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.334+0000 7f7b52800640 1 -- 172.21.15.2:0/2277522458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c069590 msgr2=0x7f7b4c105c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:09.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.334+0000 7f7b52800640 1 --2- 172.21.15.2:0/2277522458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c069590 0x7f7b4c105c40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b40009890 tx=0x7f7b4002bb70 comp rx=0 tx=0).stop 2024-03-11T18:44:09.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 -- 172.21.15.2:0/2277522458 shutdown_connections 2024-03-11T18:44:09.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 --2- 172.21.15.2:0/2277522458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c106370 0x7f7b4c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 --2- 172.21.15.2:0/2277522458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c069590 0x7f7b4c105c40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 --2- 172.21.15.2:0/2277522458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c068be0 0x7f7b4c068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 -- 172.21.15.2:0/2277522458 >> 172.21.15.2:0/2277522458 conn(0x7f7b4c0759e0 msgr2=0x7f7b4c075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:09.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 -- 172.21.15.2:0/2277522458 shutdown_connections 2024-03-11T18:44:09.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.335+0000 7f7b52800640 1 -- 172.21.15.2:0/2277522458 wait complete. 2024-03-11T18:44:09.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.337+0000 7f7b52800640 1 Processor -- start 2024-03-11T18:44:09.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.339+0000 7f7b52800640 1 -- start start 2024-03-11T18:44:09.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.340+0000 7f7b52800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c068be0 0x7f7b4c072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:09.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b52800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:09.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b52800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c06b900 0x7f7b4c06bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:09.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b52800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c06c540 con 0x7f7b4c068be0 2024-03-11T18:44:09.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b52800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c06c6b0 con 0x7f7b4c06b900 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b52800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c070350 con 0x7f7b4c106370 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b50a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b50a00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:47234/0 (socket says 172.21.15.2:47234) 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b50a00640 1 -- 172.21.15.2:0/2517502354 learned_addr learned my addr 172.21.15.2:0/2517502354 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b51400640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c068be0 0x7f7b4c072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.343+0000 7f7b51e00640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c06b900 0x7f7b4c06bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:09.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.344+0000 7f7b50a00640 1 -- 172.21.15.2:0/2517502354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c06b900 msgr2=0x7f7b4c06bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:09.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.344+0000 7f7b50a00640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c06b900 0x7f7b4c06bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.344+0000 7f7b50a00640 1 -- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c068be0 msgr2=0x7f7b4c072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:09.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.344+0000 7f7b50a00640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c068be0 0x7f7b4c072830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.345+0000 7f7b50a00640 1 -- 172.21.15.2:0/2517502354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38009490 con 0x7f7b4c106370 2024-03-11T18:44:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.345+0000 7f7b50a00640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b40037040 tx=0x7f7b400045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.346+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40034990 con 0x7f7b4c106370 2024-03-11T18:44:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.346+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b4002c9f0 con 0x7f7b4c106370 2024-03-11T18:44:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.346+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40032e60 con 0x7f7b4c106370 2024-03-11T18:44:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.346+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b40009490 con 0x7f7b4c106370 2024-03-11T18:44:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.346+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c070760 con 0x7f7b4c106370 2024-03-11T18:44:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.350+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c1026b0 con 0x7f7b4c106370 2024-03-11T18:44:09.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.353+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f7b4003b040 con 0x7f7b4c106370 2024-03-11T18:44:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.355+0000 7f7b4a000640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 0x7f7b340a4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.356+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f7b400ef770 con 0x7f7b4c106370 2024-03-11T18:44:09.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.357+0000 7f7b51400640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 0x7f7b340a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:09.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.360+0000 7f7b51400640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 0x7f7b340a4140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b44005fd0 tx=0x7f7b440095c0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:09.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.361+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b400b82a0 con 0x7f7b4c106370 2024-03-11T18:44:09.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.428 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.429 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:09 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.572 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.573 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.573 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.573 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.573 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.573 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:09 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:09.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.824+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b4c069590 con 0x7f7b340a1c80 2024-03-11T18:44:09.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.898+0000 7f7b4a000640 1 -- 172.21.15.2:0/2517502354 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f7b4c069590 con 0x7f7b340a1c80 2024-03-11T18:44:09.900 INFO:teuthology.orchestra.run.smithi002.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-11T00:00:00", "created": "2024-03-11T18:33:43", "first": "2024-03-11T18:34:00", "last": "2024-03-11T18:44:00", "last_pruned": "2024-03-11T18:44:02", "created_count": 11, "pruned_count": 5, "active": true} 2024-03-11T18:44:09.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.903+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 msgr2=0x7f7b340a4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:09.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.903+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 0x7f7b340a4140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7b44005fd0 tx=0x7f7b440095c0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.903+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 msgr2=0x7f7b4c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:09.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.903+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7b40037040 tx=0x7f7b400045b0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 shutdown_connections 2024-03-11T18:44:09.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b4c06b900 0x7f7b4c06bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4c106370 0x7f7b4c072d70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7b340a1c80 0x7f7b340a4140 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 --2- 172.21.15.2:0/2517502354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b4c068be0 0x7f7b4c072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 >> 172.21.15.2:0/2517502354 conn(0x7f7b4c0759e0 msgr2=0x7f7b4c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 shutdown_connections 2024-03-11T18:44:09.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:09.904+0000 7f7b52800640 1 -- 172.21.15.2:0/2517502354 wait complete. 2024-03-11T18:44:09.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-11T18:44:10.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:10 smithi002 ceph-mon[31328]: pgmap v327: 129 pgs: 11 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 91 active+clean; 100 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-03-11T18:44:10.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.278+0000 7f5ef4a00640 1 -- 172.21.15.2:0/2625836847 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00337c0 con 0x7f5ef80ffba0 2024-03-11T18:44:10.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 -- 172.21.15.2:0/2625836847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 msgr2=0x7f5ef810e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:10.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 --2- 172.21.15.2:0/2625836847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810e000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009c10 tx=0x7f5ef002bef0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 -- 172.21.15.2:0/2625836847 shutdown_connections 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 --2- 172.21.15.2:0/2625836847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810e000 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 --2- 172.21.15.2:0/2625836847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef80ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 --2- 172.21.15.2:0/2625836847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef8106170 0x7f5ef8106550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 -- 172.21.15.2:0/2625836847 >> 172.21.15.2:0/2625836847 conn(0x7f5ef8076170 msgr2=0x7f5ef80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:10.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.280+0000 7f5efce00640 1 -- 172.21.15.2:0/2625836847 shutdown_connections 2024-03-11T18:44:10.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.281+0000 7f5efce00640 1 -- 172.21.15.2:0/2625836847 wait complete. 2024-03-11T18:44:10.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.282+0000 7f5efce00640 1 Processor -- start 2024-03-11T18:44:10.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.284+0000 7f5efce00640 1 -- start start 2024-03-11T18:44:10.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.285+0000 7f5efce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:10.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.285+0000 7f5efce00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:10.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5efce00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 0x7f5ef81b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:10.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:10.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef5400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52742/0 (socket says 172.21.15.2:52742) 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46616/0 (socket says 172.21.15.2:46616) 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef5400640 1 -- 172.21.15.2:0/745166233 learned_addr learned my addr 172.21.15.2:0/745166233 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81b2720 con 0x7f5ef80ff1e0 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81b2860 con 0x7f5ef80ffba0 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef81b29d0 con 0x7f5ef8106170 2024-03-11T18:44:10.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.286+0000 7f5ef6800640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 0x7f5ef81b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:10.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.288+0000 7f5ef5400640 1 -- 172.21.15.2:0/745166233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 msgr2=0x7f5ef81b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:10.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.288+0000 7f5ef5400640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 0x7f5ef81b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.288+0000 7f5ef5400640 1 -- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 msgr2=0x7f5ef810d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.288+0000 7f5ef5400640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.288+0000 7f5ef5400640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef0009840 con 0x7f5ef80ffba0 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.289+0000 7f5ef5400640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400e9c0 tx=0x7f5ee400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.289+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee400ccc0 con 0x7f5ef80ffba0 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.289+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ee400ce20 con 0x7f5ef80ffba0 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.289+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4004ea0 con 0x7f5ef80ffba0 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.289+0000 7f5ef5e00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:10.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.290+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef81b2cb0 con 0x7f5ef80ffba0 2024-03-11T18:44:10.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.290+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef81b31f0 con 0x7f5ef80ffba0 2024-03-11T18:44:10.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.290+0000 7f5ef6800640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 0x7f5ef81b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:10.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.293+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef8100660 con 0x7f5ef80ffba0 2024-03-11T18:44:10.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.298+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f5ee4005590 con 0x7f5ef80ffba0 2024-03-11T18:44:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.308+0000 7f5eeea00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 0x7f5edc0a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.308+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f5ee4014070 con 0x7f5ef80ffba0 2024-03-11T18:44:10.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.310+0000 7f5ef5e00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 0x7f5edc0a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.311+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ee4096970 con 0x7f5ef80ffba0 2024-03-11T18:44:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.313+0000 7f5ef5e00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 0x7f5edc0a4220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d60 tx=0x7f5ee8009110 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:10.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:10 smithi165 ceph-mon[36634]: pgmap v327: 129 pgs: 11 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 91 active+clean; 100 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-03-11T18:44:10.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:10 smithi023 ceph-mon[36406]: pgmap v327: 129 pgs: 11 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 91 active+clean; 100 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-03-11T18:44:10.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.770+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 --> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef8111400 con 0x7f5edc0a1d60 2024-03-11T18:44:10.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.796+0000 7f5eeea00640 1 -- 172.21.15.2:0/745166233 <== mgr.25418 v2:172.21.15.2:6800/1417175579 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f5ef8111400 con 0x7f5edc0a1d60 2024-03-11T18:44:10.798 INFO:teuthology.orchestra.run.smithi002.stdout:/ 1m 6m3h 2024-03-11T18:44:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 msgr2=0x7f5edc0a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 0x7f5edc0a4220 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d60 tx=0x7f5ee8009110 comp rx=0 tx=0).stop 2024-03-11T18:44:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 msgr2=0x7f5ef810d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400e9c0 tx=0x7f5ee400edc0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 shutdown_connections 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5ef8106170 0x7f5ef81b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ef80ffba0 0x7f5ef810d6b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f5edc0a1d60 0x7f5edc0a4220 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 --2- 172.21.15.2:0/745166233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5ef80ff1e0 0x7f5ef810d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 >> 172.21.15.2:0/745166233 conn(0x7f5ef8076170 msgr2=0x7f5ef8103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 shutdown_connections 2024-03-11T18:44:10.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:10.799+0000 7f5efce00640 1 -- 172.21.15.2:0/745166233 wait complete. 2024-03-11T18:44:10.819 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-11T18:44:10.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:44:11.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.082+0000 7fac52000640 1 -- 172.21.15.2:0/2053728996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1053d0 msgr2=0x7fac4c105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.082+0000 7fac52000640 1 --2- 172.21.15.2:0/2053728996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1053d0 0x7fac4c105860 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac44009ca0 tx=0x7fac4402c1d0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 -- 172.21.15.2:0/2053728996 shutdown_connections 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 --2- 172.21.15.2:0/2053728996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1053d0 0x7fac4c105860 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 --2- 172.21.15.2:0/2053728996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac4c1008f0 0x7fac4c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 --2- 172.21.15.2:0/2053728996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c0fffd0 0x7fac4c1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 -- 172.21.15.2:0/2053728996 >> 172.21.15.2:0/2053728996 conn(0x7fac4c0fb9e0 msgr2=0x7fac4c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 -- 172.21.15.2:0/2053728996 shutdown_connections 2024-03-11T18:44:11.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.084+0000 7fac52000640 1 -- 172.21.15.2:0/2053728996 wait complete. 2024-03-11T18:44:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.085+0000 7fac52000640 1 Processor -- start 2024-03-11T18:44:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 -- start start 2024-03-11T18:44:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac4c0fffd0 0x7fac4c1a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c1053d0 0x7fac4c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c19e100 con 0x7fac4c1053d0 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c19e270 con 0x7fac4c1008f0 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.086+0000 7fac52000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c19e430 con 0x7fac4c0fffd0 2024-03-11T18:44:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52758/0 (socket says 172.21.15.2:52758) 2024-03-11T18:44:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 -- 172.21.15.2:0/2715526496 learned_addr learned my addr 172.21.15.2:0/2715526496 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac51600640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c1053d0 0x7fac4c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 -- 172.21.15.2:0/2715526496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac4c0fffd0 msgr2=0x7fac4c1a38b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-11T18:44:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac4c0fffd0 0x7fac4c1a38b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 -- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c1053d0 msgr2=0x7fac4c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c1053d0 0x7fac4c19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.087+0000 7fac4be00640 1 -- 172.21.15.2:0/2715526496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44009900 con 0x7fac4c1008f0 2024-03-11T18:44:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.089+0000 7fac4be00640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fac3800c970 tx=0x7fac3800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.090+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38007bb0 con 0x7fac4c1008f0 2024-03-11T18:44:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.090+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac38007d10 con 0x7fac4c1008f0 2024-03-11T18:44:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.090+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac38018760 con 0x7fac4c1008f0 2024-03-11T18:44:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.090+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c19e710 con 0x7fac4c1008f0 2024-03-11T18:44:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.090+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c1aa950 con 0x7fac4c1008f0 2024-03-11T18:44:11.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.092+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c04ef50 con 0x7fac4c1008f0 2024-03-11T18:44:11.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.097+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fac3800f450 con 0x7fac4c1008f0 2024-03-11T18:44:11.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.099+0000 7fac49600640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 0x7fac340a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.099+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fac380cdd20 con 0x7fac4c1008f0 2024-03-11T18:44:11.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.102+0000 7fac50c00640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 0x7fac340a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.103+0000 7fac50c00640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 0x7fac340a4220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0fee10 tx=0x7fac40006c20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.105+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac38096740 con 0x7fac4c1008f0 2024-03-11T18:44:11.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.498+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac4c19ef90 con 0x7fac4c1008f0 2024-03-11T18:44:11.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.500+0000 7fac49600640 1 -- 172.21.15.2:0/2715526496 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+25484 (secure 0 0 0) 0x7fac38099e00 con 0x7fac4c1008f0 2024-03-11T18:44:11.503 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:11.503 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":107,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:44:07.135967+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2218375343":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3512553041":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/492479102":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2651572933":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3571811795":"2024-03-12T18:33:42.314565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:44:11.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='client.26311 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.505 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:11 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 msgr2=0x7fac340a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 0x7fac340a4220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0fee10 tx=0x7fac40006c20 comp rx=0 tx=0).stop 2024-03-11T18:44:11.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 msgr2=0x7fac4c1a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fac3800c970 tx=0x7fac3800ce30 comp rx=0 tx=0).stop 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 shutdown_connections 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fac340a1d60 0x7fac340a4220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac4c1053d0 0x7fac4c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac4c1008f0 0x7fac4c1a3df0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 --2- 172.21.15.2:0/2715526496 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac4c0fffd0 0x7fac4c1a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.504+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 >> 172.21.15.2:0/2715526496 conn(0x7fac4c0fb9e0 msgr2=0x7fac4c102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:11.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.505+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 shutdown_connections 2024-03-11T18:44:11.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.505+0000 7fac52000640 1 -- 172.21.15.2:0/2715526496 wait complete. 2024-03-11T18:44:11.535 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:44:11.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='client.26311 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:11 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='client.26311 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:11 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:11.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.848+0000 7f6e73800640 1 -- 172.21.15.2:0/2493347795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c101d30 msgr2=0x7f6e6c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.848+0000 7f6e73800640 1 --2- 172.21.15.2:0/2493347795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c101d30 0x7f6e6c105d70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009900 tx=0x7f6e5802bc40 comp rx=0 tx=0).stop 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 -- 172.21.15.2:0/2493347795 shutdown_connections 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 --2- 172.21.15.2:0/2493347795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 --2- 172.21.15.2:0/2493347795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c101d30 0x7f6e6c105d70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 --2- 172.21.15.2:0/2493347795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 -- 172.21.15.2:0/2493347795 >> 172.21.15.2:0/2493347795 conn(0x7f6e6c0fcc20 msgr2=0x7f6e6c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 -- 172.21.15.2:0/2493347795 shutdown_connections 2024-03-11T18:44:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.849+0000 7f6e73800640 1 -- 172.21.15.2:0/2493347795 wait complete. 2024-03-11T18:44:11.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.850+0000 7f6e73800640 1 Processor -- start 2024-03-11T18:44:11.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 -- start start 2024-03-11T18:44:11.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c199ec0 0x7f6e6c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c19a8b0 con 0x7f6e6c106450 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c19a9f0 con 0x7f6e6c101380 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.851+0000 7f6e73800640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c19abb0 con 0x7f6e6c199ec0 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.852+0000 7f6e70e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.852+0000 7f6e70e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52772/0 (socket says 172.21.15.2:52772) 2024-03-11T18:44:11.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.852+0000 7f6e70e00640 1 -- 172.21.15.2:0/3642161055 learned_addr learned my addr 172.21.15.2:0/3642161055 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:11.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.852+0000 7f6e6be00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.852+0000 7f6e71800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c199ec0 0x7f6e6c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.853+0000 7f6e6be00640 1 -- 172.21.15.2:0/3642161055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c199ec0 msgr2=0x7f6e6c19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.853+0000 7f6e6be00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c199ec0 0x7f6e6c19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.853+0000 7f6e6be00640 1 -- 172.21.15.2:0/3642161055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 msgr2=0x7f6e6c195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.853+0000 7f6e6be00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.853+0000 7f6e6be00640 1 -- 172.21.15.2:0/3642161055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58009560 con 0x7f6e6c106450 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.854+0000 7f6e6be00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c195b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e58036040 tx=0x7f6e580040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.854+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e580044c0 con 0x7f6e6c106450 2024-03-11T18:44:11.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.854+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e580052d0 con 0x7f6e6c106450 2024-03-11T18:44:11.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.854+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58003de0 con 0x7f6e6c106450 2024-03-11T18:44:11.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.855+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c06aba0 con 0x7f6e6c106450 2024-03-11T18:44:11.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.855+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c06b030 con 0x7f6e6c106450 2024-03-11T18:44:11.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.857+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c102850 con 0x7f6e6c106450 2024-03-11T18:44:11.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.864+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f6e58004df0 con 0x7f6e6c106450 2024-03-11T18:44:11.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.866+0000 7f6e69600640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 0x7f6e500a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:11.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.867+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f6e580f02d0 con 0x7f6e6c106450 2024-03-11T18:44:11.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.868+0000 7f6e70e00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 0x7f6e500a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:11.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.872+0000 7f6e70e00640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 0x7f6e500a40e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009fd0 tx=0x7f6e64009720 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:11.880+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e580b8f60 con 0x7f6e6c106450 2024-03-11T18:44:12.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.298+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e6c19acf0 con 0x7f6e6c106450 2024-03-11T18:44:12.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.300+0000 7f6e69600640 1 -- 172.21.15.2:0/3642161055 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e580430a0 con 0x7f6e6c106450 2024-03-11T18:44:12.304 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:44:12.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.305+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 msgr2=0x7f6e500a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:12.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.305+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 0x7f6e500a40e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009fd0 tx=0x7f6e64009720 comp rx=0 tx=0).stop 2024-03-11T18:44:12.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.305+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 msgr2=0x7f6e6c195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:12.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.305+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c195b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e58036040 tx=0x7f6e580040c0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 shutdown_connections 2024-03-11T18:44:12.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e6c199ec0 0x7f6e6c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6e500a1c20 0x7f6e500a40e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e6c106450 0x7f6e6c195b90 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e58036040 tx=0x7f6e580040c0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 --2- 172.21.15.2:0/3642161055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e6c101380 0x7f6e6c195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 >> 172.21.15.2:0/3642161055 conn(0x7f6e6c0fcc20 msgr2=0x7f6e6c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 shutdown_connections 2024-03-11T18:44:12.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.306+0000 7f6e73800640 1 -- 172.21.15.2:0/3642161055 wait complete. 2024-03-11T18:44:12.334 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:44:12.448 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: pgmap v328: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:12.448 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='client.26321 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:12.448 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.448 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2715526496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.449 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:12 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.642+0000 7f63ef400640 1 -- 172.21.15.2:0/2426012030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 msgr2=0x7f63e8069c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:12.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.642+0000 7f63ef400640 1 --2- 172.21.15.2:0/2426012030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 0x7f63e8069c80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009be0 tx=0x7f63e002bf10 comp rx=0 tx=0).stop 2024-03-11T18:44:12.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.644+0000 7f63ef400640 1 -- 172.21.15.2:0/2426012030 shutdown_connections 2024-03-11T18:44:12.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.644+0000 7f63ef400640 1 --2- 172.21.15.2:0/2426012030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e8103bc0 0x7f63e8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.644+0000 7f63ef400640 1 --2- 172.21.15.2:0/2426012030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 0x7f63e8069c80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.644+0000 7f63ef400640 1 --2- 172.21.15.2:0/2426012030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e81060c0 0x7f63e80692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.645+0000 7f63ef400640 1 -- 172.21.15.2:0/2426012030 >> 172.21.15.2:0/2426012030 conn(0x7f63e8077ef0 msgr2=0x7f63e8078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:12.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.645+0000 7f63ef400640 1 -- 172.21.15.2:0/2426012030 shutdown_connections 2024-03-11T18:44:12.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.645+0000 7f63ef400640 1 -- 172.21.15.2:0/2426012030 wait complete. 2024-03-11T18:44:12.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.647+0000 7f63ef400640 1 Processor -- start 2024-03-11T18:44:12.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.647+0000 7f63ef400640 1 -- start start 2024-03-11T18:44:12.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 0x7f63e81a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:12.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e8103bc0 0x7f63e81a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:12.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:12.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e819e0c0 con 0x7f63e81060c0 2024-03-11T18:44:12.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e819e230 con 0x7f63e8069820 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.648+0000 7f63ef400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e819e3f0 con 0x7f63e8103bc0 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46676/0 (socket says 172.21.15.2:46676) 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 -- 172.21.15.2:0/93511506 learned_addr learned my addr 172.21.15.2:0/93511506 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63e7e00640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e8103bc0 0x7f63e81a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:12.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 -- 172.21.15.2:0/93511506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e8103bc0 msgr2=0x7f63e81a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:12.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e8103bc0 0x7f63e81a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 -- 172.21.15.2:0/93511506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 msgr2=0x7f63e81a38c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T18:44:12.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 0x7f63e81a38c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:12.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.649+0000 7f63ed400640 1 -- 172.21.15.2:0/93511506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d8009490 con 0x7f63e81060c0 2024-03-11T18:44:12.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.650+0000 7f63ed400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f63d800e940 tx=0x7f63d800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:12.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.651+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d800cd00 con 0x7f63e81060c0 2024-03-11T18:44:12.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.651+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d800ce60 con 0x7f63e81060c0 2024-03-11T18:44:12.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.651+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8010430 con 0x7f63e81060c0 2024-03-11T18:44:12.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.651+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e0009840 con 0x7f63e81060c0 2024-03-11T18:44:12.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.651+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e81aa780 con 0x7f63e81060c0 2024-03-11T18:44:12.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.654+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e81006a0 con 0x7f63e81060c0 2024-03-11T18:44:12.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: pgmap v328: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:12.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='client.26321 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:12.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2715526496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.658 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:12 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: pgmap v328: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:12.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='client.26321 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T18:44:12.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi165 to 174.1M 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi165 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2715526496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.659 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.660 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:12 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:12.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.656+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f63d8004020 con 0x7f63e81060c0 2024-03-11T18:44:12.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.659+0000 7f63e5600640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 0x7f63cc0a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:12.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.659+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f63d8014070 con 0x7f63e81060c0 2024-03-11T18:44:12.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.662+0000 7f63eca00640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 0x7f63cc0a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:12.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.663+0000 7f63eca00640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 0x7f63cc0a4160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f63e819ebb0 tx=0x7f63d4009260 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:12.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:12.670+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f63d8096b90 con 0x7f63e81060c0 2024-03-11T18:44:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.063+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63e8117830 con 0x7f63e81060c0 2024-03-11T18:44:13.072 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:44:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.067+0000 7f63e5600640 1 -- 172.21.15.2:0/93511506 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63d809a250 con 0x7f63e81060c0 2024-03-11T18:44:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.069+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 msgr2=0x7f63cc0a4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.069+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 0x7f63cc0a4160 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f63e819ebb0 tx=0x7f63d4009260 comp rx=0 tx=0).stop 2024-03-11T18:44:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.069+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 msgr2=0x7f63e819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.069+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f63d800e940 tx=0x7f63d800ee00 comp rx=0 tx=0).stop 2024-03-11T18:44:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 shutdown_connections 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f63cc0a1ca0 0x7f63cc0a4160 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f63e81060c0 0x7f63e819da70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63e8103bc0 0x7f63e81a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 --2- 172.21.15.2:0/93511506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63e8069820 0x7f63e81a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 >> 172.21.15.2:0/93511506 conn(0x7f63e8077ef0 msgr2=0x7f63e8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 shutdown_connections 2024-03-11T18:44:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.071+0000 7f63ef400640 1 -- 172.21.15.2:0/93511506 wait complete. 2024-03-11T18:44:13.103 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 -- 172.21.15.2:0/3085389877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 msgr2=0x7f18b4075a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 --2- 172.21.15.2:0/3085389877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b4075a90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f18a4009900 tx=0x7f18a402bc30 comp rx=0 tx=0).stop 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 -- 172.21.15.2:0/3085389877 shutdown_connections 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 --2- 172.21.15.2:0/3085389877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 0x7f18b41143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 --2- 172.21.15.2:0/3085389877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b4111a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 --2- 172.21.15.2:0/3085389877 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b4075a90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 -- 172.21.15.2:0/3085389877 >> 172.21.15.2:0/3085389877 conn(0x7f18b40fedd0 msgr2=0x7f18b41011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:13.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 -- 172.21.15.2:0/3085389877 shutdown_connections 2024-03-11T18:44:13.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.312+0000 7f18bb000640 1 -- 172.21.15.2:0/3085389877 wait complete. 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.313+0000 7f18bb000640 1 Processor -- start 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.313+0000 7f18bb000640 1 -- start start 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b419e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 0x7f18b41a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41a2c50 con 0x7f18b40756b0 2024-03-11T18:44:13.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41a2dc0 con 0x7f18b4075fd0 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18bb000640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41a2f80 con 0x7f18b4111fc0 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b419e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46694/0 (socket says 172.21.15.2:46694) 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 -- 172.21.15.2:0/1758019642 learned_addr learned my addr 172.21.15.2:0/1758019642 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b9000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 0x7f18b41a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 -- 172.21.15.2:0/1758019642 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 msgr2=0x7f18b41a2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 0x7f18b41a2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 -- 172.21.15.2:0/1758019642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 msgr2=0x7f18b419e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b419e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3e00640 1 -- 172.21.15.2:0/1758019642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009560 con 0x7f18b40756b0 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.314+0000 7f18b3400640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b419e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18b3e00640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f18a4000c00 tx=0x7f18a402c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a402dea0 con 0x7f18b40756b0 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18a4004410 con 0x7f18b40756b0 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b41a3200 con 0x7f18b40756b0 2024-03-11T18:44:13.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4003c20 con 0x7f18b40756b0 2024-03-11T18:44:13.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.315+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b41aaae0 con 0x7f18b40756b0 2024-03-11T18:44:13.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.316+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4076df0 con 0x7f18b40756b0 2024-03-11T18:44:13.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.318+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f18a40054f0 con 0x7f18b40756b0 2024-03-11T18:44:13.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.319+0000 7f18b0c00640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 0x7f18980a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.320+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f18a40f01a0 con 0x7f18b40756b0 2024-03-11T18:44:13.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.320+0000 7f18b3400640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 0x7f18980a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.320+0000 7f18b3400640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 0x7f18980a4060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18b41012a0 tx=0x7f18a0005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:13.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.324+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18a40b8cd0 con 0x7f18b40756b0 2024-03-11T18:44:13.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:44:13.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:44:13.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3642161055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/93511506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:13 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.517+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f18b410d150 con 0x7f18b40756b0 2024-03-11T18:44:13.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.517+0000 7f18b0c00640 1 -- 172.21.15.2:0/1758019642 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18a40bc390 con 0x7f18b40756b0 2024-03-11T18:44:13.520 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:44:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 msgr2=0x7f18980a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 0x7f18980a4060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18b41012a0 tx=0x7f18a0005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 msgr2=0x7f18b419dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f18a4000c00 tx=0x7f18a402c6b0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 shutdown_connections 2024-03-11T18:44:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f18b4111fc0 0x7f18b41a2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18b4075fd0 0x7f18b419e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f18980a1ba0 0x7f18980a4060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 --2- 172.21.15.2:0/1758019642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18b40756b0 0x7f18b419dd90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 >> 172.21.15.2:0/1758019642 conn(0x7f18b40fedd0 msgr2=0x7f18b4113a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 shutdown_connections 2024-03-11T18:44:13.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.520+0000 7f18bb000640 1 -- 172.21.15.2:0/1758019642 wait complete. 2024-03-11T18:44:13.539 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:44:13.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:44:13.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:44:13.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3642161055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/93511506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:13 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi023 to 174.1M 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi023 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3642161055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/93511506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:13.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.658 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:13 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:13.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 -- 172.21.15.2:0/4032470362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 msgr2=0x7f6f7c10e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 --2- 172.21.15.2:0/4032470362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c10e020 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009c10 tx=0x7f6f7402bee0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 -- 172.21.15.2:0/4032470362 shutdown_connections 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 --2- 172.21.15.2:0/4032470362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c10e020 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 --2- 172.21.15.2:0/4032470362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c101d30 0x7f6f7c105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 --2- 172.21.15.2:0/4032470362 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 0x7f6f7c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 -- 172.21.15.2:0/4032470362 >> 172.21.15.2:0/4032470362 conn(0x7f6f7c0fcc00 msgr2=0x7f6f7c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 -- 172.21.15.2:0/4032470362 shutdown_connections 2024-03-11T18:44:13.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.738+0000 7f6f81600640 1 -- 172.21.15.2:0/4032470362 wait complete. 2024-03-11T18:44:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.739+0000 7f6f81600640 1 Processor -- start 2024-03-11T18:44:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.739+0000 7f6f81600640 1 -- start start 2024-03-11T18:44:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 0x7f6f7c199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c19e3d0 0x7f6f7c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c19edc0 con 0x7f6f7c101380 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c19ef30 con 0x7f6f7c106450 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f81600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c19f0f0 con 0x7f6f7c19e3d0 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:52818/0 (socket says 172.21.15.2:52818) 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 -- 172.21.15.2:0/1786051450 learned_addr learned my addr 172.21.15.2:0/1786051450 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7aa00640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 0x7f6f7c199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7b400640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c19e3d0 0x7f6f7c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 -- 172.21.15.2:0/1786051450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c19e3d0 msgr2=0x7f6f7c19e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c19e3d0 0x7f6f7c19e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 -- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 msgr2=0x7f6f7c199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 0x7f6f7c199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.740+0000 7f6f7a000640 1 -- 172.21.15.2:0/1786051450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f74009840 con 0x7f6f7c106450 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f7a000640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800e9f0 tx=0x7f6f6800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f680096c0 con 0x7f6f7c106450 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f68005500 con 0x7f6f7c106450 2024-03-11T18:44:13.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c1a6680 con 0x7f6f7c106450 2024-03-11T18:44:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f68010430 con 0x7f6f7c106450 2024-03-11T18:44:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.741+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c1a6ae0 con 0x7f6f7c106450 2024-03-11T18:44:13.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.743+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c067770 con 0x7f6f7c106450 2024-03-11T18:44:13.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.748+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f6f68005020 con 0x7f6f7c106450 2024-03-11T18:44:13.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.750+0000 7f6f73400640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 0x7f6f600a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:13.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.750+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7f6f68014070 con 0x7f6f7c106450 2024-03-11T18:44:13.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.750+0000 7f6f7aa00640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 0x7f6f600a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:13.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.751+0000 7f6f7aa00640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 0x7f6f600a4060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005fd0 tx=0x7f6f6c009260 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:13.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.754+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6f68096940 con 0x7f6f7c106450 2024-03-11T18:44:13.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.959+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6f7c101d30 con 0x7f6f7c106450 2024-03-11T18:44:13.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.960+0000 7f6f73400640 1 -- 172.21.15.2:0/1786051450 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6f6809a000 con 0x7f6f7c106450 2024-03-11T18:44:13.962 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 msgr2=0x7f6f600a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 0x7f6f600a4060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005fd0 tx=0x7f6f6c009260 comp rx=0 tx=0).stop 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 msgr2=0x7f6f7c19a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800e9f0 tx=0x7f6f6800edf0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 shutdown_connections 2024-03-11T18:44:13.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7c19e3d0 0x7f6f7c19e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f7c106450 0x7f6f7c19a020 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f6f600a1ba0 0x7f6f600a4060 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 --2- 172.21.15.2:0/1786051450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f7c101380 0x7f6f7c199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.963+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 >> 172.21.15.2:0/1786051450 conn(0x7f6f7c0fcc00 msgr2=0x7f6f7c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.964+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 shutdown_connections 2024-03-11T18:44:13.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:13.964+0000 7f6f81600640 1 -- 172.21.15.2:0/1786051450 wait complete. 2024-03-11T18:44:13.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-11T18:44:14.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 -- 172.21.15.2:0/3358818755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb881021b0 msgr2=0x7fcb88102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:14.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 --2- 172.21.15.2:0/3358818755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb881021b0 0x7fcb88102590 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009ca0 tx=0x7fcb7802c190 comp rx=0 tx=0).stop 2024-03-11T18:44:14.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 -- 172.21.15.2:0/3358818755 shutdown_connections 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 --2- 172.21.15.2:0/3358818755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb8810b580 0x7fcb8810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 --2- 172.21.15.2:0/3358818755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8810afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 --2- 172.21.15.2:0/3358818755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb881021b0 0x7fcb88102590 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 -- 172.21.15.2:0/3358818755 >> 172.21.15.2:0/3358818755 conn(0x7fcb88076c10 msgr2=0x7fcb880fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 -- 172.21.15.2:0/3358818755 shutdown_connections 2024-03-11T18:44:14.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.182+0000 7fcb8da00640 1 -- 172.21.15.2:0/3358818755 wait complete. 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.183+0000 7fcb8da00640 1 Processor -- start 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.183+0000 7fcb8da00640 1 -- start start 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.183+0000 7fcb8da00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb881021b0 0x7fcb8819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8da00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8819f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8da00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88199c30 con 0x7fcb8810b580 2024-03-11T18:44:14.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8da00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88199da0 con 0x7fcb881021b0 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8da00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb88199f60 con 0x7fcb88102ad0 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46724/0 (socket says 172.21.15.2:46724) 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 -- 172.21.15.2:0/2646592217 learned_addr learned my addr 172.21.15.2:0/2646592217 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb87e00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb881021b0 0x7fcb8819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb7f400640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8819f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 -- 172.21.15.2:0/2646592217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 msgr2=0x7fcb8819f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8819f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:14.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 -- 172.21.15.2:0/2646592217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb881021b0 msgr2=0x7fcb8819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:14.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb881021b0 0x7fcb8819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:14.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb8d000640 1 -- 172.21.15.2:0/2646592217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78009900 con 0x7fcb8810b580 2024-03-11T18:44:14.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.184+0000 7fcb7f400640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:44:14.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.185+0000 7fcb8d000640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e970 tx=0x7fcb8000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:14.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.188+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80009720 con 0x7fcb8810b580 2024-03-11T18:44:14.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.188+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb8819a240 con 0x7fcb8810b580 2024-03-11T18:44:14.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.188+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb80005500 con 0x7fcb8810b580 2024-03-11T18:44:14.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.188+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80010430 con 0x7fcb8810b580 2024-03-11T18:44:14.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.188+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb881a64d0 con 0x7fcb8810b580 2024-03-11T18:44:14.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.190+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb8804eee0 con 0x7fcb8810b580 2024-03-11T18:44:14.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.191+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fcb80005020 con 0x7fcb8810b580 2024-03-11T18:44:14.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.193+0000 7fcb85600640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 0x7fcb6c0a4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:14.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.193+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9664+0+0 (secure 0 0 0) 0x7fcb80014070 con 0x7fcb8810b580 2024-03-11T18:44:14.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.193+0000 7fcb87e00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 0x7fcb6c0a4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:14.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.194+0000 7fcb87e00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 0x7fcb6c0a4040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000f80 tx=0x7fcb78005f10 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:14.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.201+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcb80096aa0 con 0x7fcb8810b580 2024-03-11T18:44:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:14 smithi002 ceph-mon[31328]: pgmap v329: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:14 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1758019642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:14.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:14 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1786051450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:14.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.526+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fcb8819ab10 con 0x7fcb8810b580 2024-03-11T18:44:14.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.558+0000 7fcb85600640 1 -- 172.21.15.2:0/2646592217 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+734991 (secure 0 0 0) 0x7fcb8009a160 con 0x7fcb8810b580 2024-03-11T18:44:14.571 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:14.571 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:44:14.571 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-03-11T18:44:14.571 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "active_gid": 25418, 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "active_name": "x", 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addrs": { 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6800", 2024-03-11T18:44:14.572 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1417175579 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:6801", 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1417175579 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.573 INFO:teuthology.orchestra.run.smithi002.stdout: "active_addr": "172.21.15.2:6801/1417175579", 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "active_change": "2024-03-11T18:33:42.314886+0000", 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "active_mgr_features": 4540701547738038271, 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "standbys": [ 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25412, 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "y", 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540701547738038271, 2024-03-11T18:44:14.574 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.576 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.577 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.578 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.579 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.580 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.581 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.582 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.583 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.585 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.586 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.587 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.589 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.590 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.591 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2024-03-11T18:44:14.592 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.593 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.594 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.595 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.596 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.597 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.598 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.599 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2024-03-11T18:44:14.600 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T18:44:14.601 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.602 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.603 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.605 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.606 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2024-03-11T18:44:14.607 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2024-03-11T18:44:14.608 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:14.609 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.610 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.612 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.613 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.614 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.615 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.616 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.617 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.618 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.619 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:14.620 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.621 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.622 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "read", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap-read" 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.623 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.624 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.625 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.625 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.625 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T18:44:14.625 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.626 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.626 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.626 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:14.627 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:14.627 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.627 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.627 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.627 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:14.628 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.628 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.628 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.628 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T18:44:14.629 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.629 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.629 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.630 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2024-03-11T18:44:14.630 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2024-03-11T18:44:14.630 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.630 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.631 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.631 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:14.631 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:14.631 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.632 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.632 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T18:44:14.632 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T18:44:14.632 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.632 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.633 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2024-03-11T18:44:14.633 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2024-03-11T18:44:14.633 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:14.633 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.634 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.634 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.634 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.635 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.635 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T18:44:14.635 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.635 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.635 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.636 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.636 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.636 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.636 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.636 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2024-03-11T18:44:14.637 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:14.638 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.638 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:14.638 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.638 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.639 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.639 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T18:44:14.639 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.639 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.639 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.640 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2024-03-11T18:44:14.640 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2024-03-11T18:44:14.640 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2024-03-11T18:44:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.642 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.642 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T18:44:14.642 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.643 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T18:44:14.644 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.645 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.645 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.645 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2024-03-11T18:44:14.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2024-03-11T18:44:14.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.650 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm_log_destination": { 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm_log_destination", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "file", 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "file,syslog", 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "syslog" 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.652 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.653 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2024-03-11T18:44:14.654 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T18:44:14.655 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.656 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.658 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2024-03-11T18:44:14.659 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.661 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.662 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.663 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.667 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.668 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T18:44:14.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2024-03-11T18:44:14.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2024-03-11T18:44:14.673 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.674 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.675 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_nvmeof": { 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_nvmeof", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.676 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Nvme-of container image", 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.677 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.678 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.679 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.680 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.681 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2024-03-11T18:44:14.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.682 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.686 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2024-03-11T18:44:14.687 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T18:44:14.688 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.689 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.690 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2024-03-11T18:44:14.691 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2024-03-11T18:44:14.692 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.693 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.694 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.695 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:14.696 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.697 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.698 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "hw_monitoring": { 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hw_monitoring", 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.699 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.700 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "inventory_list_all": { 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "inventory_list_all", 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.701 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.702 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.703 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.704 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.705 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.707 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.708 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.709 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.711 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.712 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.713 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.714 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.715 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.716 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.717 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.718 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.720 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "oob_default_addr": { 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "oob_default_addr", 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "169.254.1.1", 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.722 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.723 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.724 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.725 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2024-03-11T18:44:14.726 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.727 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.728 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.729 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.730 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.731 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.732 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.733 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.734 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.735 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.736 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2024-03-11T18:44:14.737 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T18:44:14.738 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.739 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.740 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.741 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.742 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.743 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.745 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.746 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.747 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.749 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.750 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.751 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.752 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.753 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.754 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.755 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.757 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.758 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.759 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T18:44:14.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.761 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.762 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.763 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.764 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T18:44:14.765 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.766 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.767 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.768 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.769 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T18:44:14.770 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.771 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.772 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.773 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.774 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.775 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.776 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.777 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.778 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.779 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.780 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T18:44:14.781 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.782 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.783 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T18:44:14.784 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.785 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.786 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T18:44:14.787 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.788 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.789 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.790 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.791 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.792 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.794 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.795 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.796 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.797 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T18:44:14.798 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.799 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.800 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T18:44:14.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.802 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.803 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T18:44:14.804 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.805 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.806 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T18:44:14.807 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.808 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.809 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T18:44:14.810 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.811 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.812 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T18:44:14.813 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.814 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:14.815 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.816 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.817 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.818 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.819 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:14.820 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T18:44:14.821 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.822 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.823 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.825 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2024-03-11T18:44:14.826 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.827 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.828 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.829 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.830 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.831 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.832 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.833 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.834 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.835 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.836 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.837 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.838 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.839 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.840 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.841 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.842 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.843 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.844 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.845 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2024-03-11T18:44:14.846 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.847 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.848 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.849 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.850 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2024-03-11T18:44:14.851 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.852 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:14.853 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.854 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.855 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.856 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.857 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.858 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.859 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.860 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.861 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.862 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.863 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.864 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.865 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2024-03-11T18:44:14.866 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T18:44:14.867 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.868 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2024-03-11T18:44:14.869 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.870 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.871 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.872 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.873 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.874 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.875 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.876 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2024-03-11T18:44:14.877 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.878 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.879 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.880 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.882 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.883 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.884 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2024-03-11T18:44:14.885 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.886 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.889 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.890 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.891 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.892 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.893 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.894 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.895 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.896 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.897 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.898 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.899 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.900 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.902 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.903 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.904 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.905 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.906 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.907 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:14.908 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:14.909 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.910 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.911 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.912 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.913 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.914 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.915 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.916 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.917 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.918 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.919 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.920 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.921 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.923 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.924 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.925 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.926 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.927 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.928 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.929 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.931 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2024-03-11T18:44:14.932 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2024-03-11T18:44:14.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.934 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.935 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.936 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.937 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.938 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.939 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.940 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.941 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.942 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.943 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.944 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.946 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.947 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.949 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.950 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.951 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.952 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.953 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.954 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.955 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.956 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2024-03-11T18:44:14.957 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.958 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.959 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2024-03-11T18:44:14.960 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.961 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.962 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.963 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.964 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.965 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.968 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.969 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.970 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.972 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.975 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.976 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.979 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.980 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.981 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.982 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.983 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.984 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.985 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.986 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.987 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.988 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.989 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.990 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:14.993 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.994 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.996 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:14.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2024-03-11T18:44:14.998 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:14.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.000 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.001 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.002 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.003 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.004 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.005 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.006 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.007 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2024-03-11T18:44:15.008 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.009 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.010 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.011 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.012 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.014 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.015 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.016 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.017 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.018 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.019 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.020 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.021 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.022 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.024 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.025 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.026 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:15.027 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.029 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T18:44:15.030 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.031 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.032 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.033 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.034 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.035 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.036 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.037 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.038 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.039 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.040 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.041 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:15.042 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.044 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.045 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.046 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.047 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.048 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.049 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.050 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.051 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.052 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.053 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.054 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.055 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.056 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.057 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.058 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.059 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.060 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.061 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.062 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2024-03-11T18:44:15.063 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.064 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2024-03-11T18:44:15.065 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.066 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:15.067 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.069 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.070 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.071 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.072 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.073 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.074 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.076 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.077 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.080 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.081 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.082 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.083 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:15.084 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.085 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.086 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.087 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.088 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.089 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.090 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.091 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.094 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.095 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:15.096 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.097 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.098 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.099 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.101 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.104 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.105 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.106 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.107 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.108 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.109 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.110 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.111 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.112 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.113 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.114 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.115 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.116 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.117 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.118 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.119 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.120 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.121 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.122 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.123 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.124 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.125 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2024-03-11T18:44:15.126 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.127 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.128 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2024-03-11T18:44:15.129 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.130 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.131 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.132 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.133 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.134 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.135 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.136 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.137 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.138 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:15.139 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2024-03-11T18:44:15.140 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.141 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.142 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.143 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.144 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.145 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.146 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.148 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.149 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.150 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.151 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.155 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.156 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.157 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.158 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.159 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.160 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.161 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.162 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.163 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.164 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.165 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.166 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.167 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.168 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.169 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.171 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.172 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.173 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.174 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.175 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.176 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.177 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.178 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.179 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.180 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.181 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.183 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.185 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.189 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2024-03-11T18:44:15.190 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T18:44:15.191 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.192 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2024-03-11T18:44:15.193 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T18:44:15.194 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.195 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.196 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.197 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2024-03-11T18:44:15.197 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2024-03-11T18:44:15.197 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.197 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.198 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2024-03-11T18:44:15.199 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2024-03-11T18:44:15.200 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.200 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.200 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.200 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T18:44:15.200 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.201 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.202 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.203 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.204 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.205 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.206 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.208 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.209 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.210 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.211 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.212 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.213 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.214 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.215 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.216 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.217 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.218 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.219 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.220 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.221 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.222 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.223 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.224 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.225 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.227 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.228 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.229 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.230 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.231 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.233 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.234 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.235 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.236 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.237 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.238 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.239 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.240 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2024-03-11T18:44:15.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "periodic_async_work": { 2024-03-11T18:44:15.242 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "periodic_async_work", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Periodically check for async work", 2024-03-11T18:44:15.243 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.244 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2024-03-11T18:44:15.246 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.247 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.248 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.249 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.250 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.251 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.252 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.253 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.254 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.255 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.256 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.257 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.258 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.259 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.260 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.262 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.263 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 25417, 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr_features": 4540701547738038271, 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.264 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.265 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.266 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.267 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.268 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.269 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.270 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.271 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.273 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.274 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2024-03-11T18:44:15.275 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.276 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.277 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2024-03-11T18:44:15.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2024-03-11T18:44:15.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.280 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.281 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.282 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.283 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.286 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.288 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.289 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.290 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:15.291 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:15.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2024-03-11T18:44:15.293 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T18:44:15.294 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2024-03-11T18:44:15.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2024-03-11T18:44:15.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.298 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.299 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.300 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.301 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.302 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.305 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.306 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:15.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2024-03-11T18:44:15.312 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "read", 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap", 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap-read" 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.313 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.314 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.315 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.320 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2024-03-11T18:44:15.321 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.322 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.323 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.324 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2024-03-11T18:44:15.325 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.326 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T18:44:15.327 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.330 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm_log_destination": { 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm_log_destination", 2024-03-11T18:44:15.331 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "file", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "file,syslog", 2024-03-11T18:44:15.332 INFO:teuthology.orchestra.run.smithi002.stdout: "syslog" 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2024-03-11T18:44:15.333 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.334 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.335 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2024-03-11T18:44:15.336 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.337 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.338 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.339 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.340 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2024-03-11T18:44:15.341 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2024-03-11T18:44:15.342 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.343 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.345 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2024-03-11T18:44:15.346 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2024-03-11T18:44:15.347 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.348 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.350 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2024-03-11T18:44:15.351 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2024-03-11T18:44:15.352 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.353 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.354 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.355 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_nvmeof": { 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_nvmeof", 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.356 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Nvme-of container image", 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.357 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T18:44:15.358 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2024-03-11T18:44:15.359 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2024-03-11T18:44:15.360 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.361 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.362 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.363 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.364 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.365 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2024-03-11T18:44:15.366 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2024-03-11T18:44:15.367 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T18:44:15.368 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.369 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.370 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.371 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2024-03-11T18:44:15.372 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2024-03-11T18:44:15.373 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.374 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "hw_monitoring": { 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hw_monitoring", 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.376 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "inventory_list_all": { 2024-03-11T18:44:15.377 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "inventory_list_all", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T18:44:15.378 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.379 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.380 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.381 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.382 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.383 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.384 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.385 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.386 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.387 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.388 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.389 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T18:44:15.390 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T18:44:15.391 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.392 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.393 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.394 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.395 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.397 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "oob_default_addr": { 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "oob_default_addr", 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "169.254.1.1", 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.399 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2024-03-11T18:44:15.400 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.401 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.402 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.403 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.404 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2024-03-11T18:44:15.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.406 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.407 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2024-03-11T18:44:15.408 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T18:44:15.409 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.410 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.411 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.412 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2024-03-11T18:44:15.413 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T18:44:15.414 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.415 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.416 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T18:44:15.417 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.418 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.420 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.421 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.422 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.423 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.424 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.425 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.426 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.427 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.428 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.429 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.432 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.433 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T18:44:15.434 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.435 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.436 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T18:44:15.437 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.441 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T18:44:15.442 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.444 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T18:44:15.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.448 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.449 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.450 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.451 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T18:44:15.452 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.453 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.454 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.455 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.456 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T18:44:15.457 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.459 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.460 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.461 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T18:44:15.462 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.463 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.467 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.468 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.469 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.470 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.471 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T18:44:15.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.473 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.474 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T18:44:15.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.478 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.479 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.480 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.481 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T18:44:15.482 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.483 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.484 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.486 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.487 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.488 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T18:44:15.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.492 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.494 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T18:44:15.495 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.496 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.497 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.498 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.500 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.501 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.502 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T18:44:15.503 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.506 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.507 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.508 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.510 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:15.511 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.512 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.513 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.514 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.515 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.516 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.517 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.518 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.519 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.520 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.521 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.522 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.523 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.524 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.525 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.526 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.527 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.528 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2024-03-11T18:44:15.529 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.530 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.531 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.532 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.533 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.534 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.535 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.536 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2024-03-11T18:44:15.537 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2024-03-11T18:44:15.538 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.539 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.540 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.541 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.542 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.543 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.544 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.546 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2024-03-11T18:44:15.547 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.548 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2024-03-11T18:44:15.549 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.550 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.552 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.553 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.554 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T18:44:15.556 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.557 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.558 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.559 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.560 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.561 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.562 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.564 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.565 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.566 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.567 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:15.568 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.569 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.570 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.571 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.572 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2024-03-11T18:44:15.573 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.574 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.576 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.577 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.578 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.579 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.580 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.581 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.582 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.583 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.584 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.585 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.587 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.588 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.590 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:15.591 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.592 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.593 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.594 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.595 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2024-03-11T18:44:15.596 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.597 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.598 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.599 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.600 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.601 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.602 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.603 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.605 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.606 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.607 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.608 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.609 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.610 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.611 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.612 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.613 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.614 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.615 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.617 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2024-03-11T18:44:15.618 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.619 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.620 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.621 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.622 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.623 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.624 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.625 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.626 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.627 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.628 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.629 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.631 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.632 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.633 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.634 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.635 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.636 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.637 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.638 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.640 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.641 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.642 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.643 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.645 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.648 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.649 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.650 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.653 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.654 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.655 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2024-03-11T18:44:15.657 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.658 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.659 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.660 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.661 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.662 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.663 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.664 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.665 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.666 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.667 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.668 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.669 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.670 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.671 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.672 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.673 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.674 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.675 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.676 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.677 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.678 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.679 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.680 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.681 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.682 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.683 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.684 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.685 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.686 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.687 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.688 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.689 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2024-03-11T18:44:15.690 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.691 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.692 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.694 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.696 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.697 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.698 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.699 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.700 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.701 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.702 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.703 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.704 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.705 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.706 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.707 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.708 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.709 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.710 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.711 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.712 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.713 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.714 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.715 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.716 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.717 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.718 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.719 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.720 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.721 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.722 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2024-03-11T18:44:15.723 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2024-03-11T18:44:15.724 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.725 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.726 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T18:44:15.727 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.728 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.729 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.730 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.731 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.732 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.733 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.734 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.735 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.736 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.738 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.739 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2024-03-11T18:44:15.740 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.741 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.742 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.743 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2024-03-11T18:44:15.744 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.745 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2024-03-11T18:44:15.746 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T18:44:15.747 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.748 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.749 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.750 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.751 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.752 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.753 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.754 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.755 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.756 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.757 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.758 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.759 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.760 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.761 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.762 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:15.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.764 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:15.765 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.766 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2024-03-11T18:44:15.767 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.768 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.769 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2024-03-11T18:44:15.770 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:15.771 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.772 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.773 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.774 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.775 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.776 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.777 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.778 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.779 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.780 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.781 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.782 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2024-03-11T18:44:15.783 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.784 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.785 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.786 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.787 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.788 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.789 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.790 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.792 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.793 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.794 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.795 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.796 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.797 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.798 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.799 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.800 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.801 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.802 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.803 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.804 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.805 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.806 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.807 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.808 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.809 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.810 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.811 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.812 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.813 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.814 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.815 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.816 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.817 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.818 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.819 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.820 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.822 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.823 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.824 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.825 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.826 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.826 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.826 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.826 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.826 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.827 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.827 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.827 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.827 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.827 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.828 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2024-03-11T18:44:15.828 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2024-03-11T18:44:15.828 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.828 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.828 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.829 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2024-03-11T18:44:15.829 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.829 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.829 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.830 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T18:44:15.830 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.830 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.830 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.830 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.831 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.831 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.831 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.831 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2024-03-11T18:44:15.832 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.832 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.832 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.832 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.832 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.833 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.834 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.835 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.836 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.837 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.838 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.839 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.840 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.841 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.842 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.843 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.844 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.845 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.846 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2024-03-11T18:44:15.847 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.848 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2024-03-11T18:44:15.849 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.850 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2024-03-11T18:44:15.851 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.852 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.853 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.854 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.855 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.856 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.857 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.858 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.859 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.860 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.861 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2024-03-11T18:44:15.862 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.863 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2024-03-11T18:44:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.865 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.866 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.867 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.868 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.869 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.870 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.871 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.872 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.873 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.874 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.875 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.876 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.877 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.878 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.879 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.880 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.881 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.882 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.883 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.884 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.885 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.886 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.887 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.888 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.889 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.890 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.891 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.892 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.893 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.894 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.895 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.896 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.897 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2024-03-11T18:44:15.898 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.899 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.900 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2024-03-11T18:44:15.901 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.902 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.903 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.904 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.905 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.906 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.909 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.911 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.912 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.913 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T18:44:15.914 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.915 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.916 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.917 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.918 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.919 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.920 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.921 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.922 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.923 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.924 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.925 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2024-03-11T18:44:15.926 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.929 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.930 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.931 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2024-03-11T18:44:15.932 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.933 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.934 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.935 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.936 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.937 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.947 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.948 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.949 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.950 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.951 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.952 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.953 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.954 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.955 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.956 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.957 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.958 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.959 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.960 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.961 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.962 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.963 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.964 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.965 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.966 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.967 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.968 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.969 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.971 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.972 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.973 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.974 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "periodic_async_work": { 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "periodic_async_work", 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.975 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Periodically check for async work", 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.976 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.977 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.978 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.979 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.980 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.981 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.982 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.983 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.984 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.985 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.986 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.987 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.988 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.989 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:15.990 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:15.991 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.992 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.993 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.994 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:15.995 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.996 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:15.997 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:15.998 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:15.999 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard", 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "restful", 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: "snap_schedule" 2024-03-11T18:44:16.000 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "available_modules": [ 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "alerts", 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:16.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.002 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How frequently to reexamine health status", 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.003 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.004 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.005 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.006 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.007 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.008 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.009 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.010 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.011 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_destination": { 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_destination", 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.012 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email address to send alerts to", 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_from_name": { 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_from_name", 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "Ceph", 2024-03-11T18:44:16.014 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Email From: name", 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_host": { 2024-03-11T18:44:16.015 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_host", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.016 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP server", 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_password": { 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_password", 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.017 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Password to authenticate with", 2024-03-11T18:44:16.018 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_port": { 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_port", 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.019 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "465", 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP port", 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.020 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_sender": { 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_sender", 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.021 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SMTP envelope sender", 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.022 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_ssl": { 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_ssl", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.023 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "smtp_user": { 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "smtp_user", 2024-03-11T18:44:16.024 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "User to authenticate as", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.025 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "balancer", 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.026 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "active": { 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "active", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.027 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically balance PGs across cluster", 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_time": { 2024-03-11T18:44:16.028 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_time", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0000", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.029 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "beginning time of day to automatically balance", 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "begin_weekday": { 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "begin_weekday", 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:16.030 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:16.031 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_max_iterations": { 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_max_iterations", 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "25", 2024-03-11T18:44:16.032 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "250", 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.033 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_metrics": { 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_metrics", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "pgs,objects,bytes", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.034 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "crush_compat_step": { 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crush_compat_step", 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.035 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.5", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0.001", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "0.999", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "aggressiveness of optimization", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-11T18:44:16.036 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "end_time": { 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_time", 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2359", 2024-03-11T18:44:16.037 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ending time of day to automatically balance", 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.038 INFO:teuthology.orchestra.run.smithi002.stdout: "end_weekday": { 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "end_weekday", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "0", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "6", 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.039 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.040 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.041 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.042 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.043 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.044 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.045 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.046 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.047 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.048 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "min_score": { 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_score", 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.049 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.050 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.051 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "upmap", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "crush-compat", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "none", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "read", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap", 2024-03-11T18:44:16.052 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap-read" 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Balancer mode", 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_ids": { 2024-03-11T18:44:16.053 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_ids", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.054 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:16.055 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.056 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_deviation": { 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_deviation", 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.057 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-11T18:44:16.058 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "upmap_max_optimizations": { 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "upmap_max_optimizations", 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.059 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.060 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm", 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.061 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_down_multiplier": { 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_down_multiplier", 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.062 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_refresh_rate": { 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_refresh_rate", 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "20", 2024-03-11T18:44:16.064 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.065 INFO:teuthology.orchestra.run.smithi002.stdout: "agent_starting_port": { 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "agent_starting_port", 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4721", 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.066 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.067 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_ptrace": { 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_ptrace", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.068 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_interval": { 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_interval", 2024-03-11T18:44:16.069 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.070 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.072 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:14 smithi023 ceph-mon[36406]: pgmap v329: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:14 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1758019642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:14 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1786051450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.073 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:15 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2646592217' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:14 smithi165 ceph-mon[36634]: pgmap v329: 129 pgs: 19 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 55 active+clean; 99 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 1.8 MiB/s wr, 248 op/s 2024-03-11T18:44:16.074 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:14 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1758019642' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:14 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1786051450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.075 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.076 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.076 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:44:16.076 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:44:16.076 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:16.076 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:15 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2646592217' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "autotune_memory_target_ratio": { 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "autotune_memory_target_ratio", 2024-03-11T18:44:16.078 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0.7", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-11T18:44:16.079 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm_log_destination": { 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephadm_log_destination", 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.080 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "file", 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "file,syslog", 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "syslog" 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.081 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "cgroups_split": { 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cgroups_split", 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.082 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.083 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "config_checks_enabled": { 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_checks_enabled", 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.084 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "config_dashboard": { 2024-03-11T18:44:16.085 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_dashboard", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-11T18:44:16.086 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_alertmanager": { 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_alertmanager", 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.087 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.088 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_base": { 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_base", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.089 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Container image name, without the tag", 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_elasticsearch": { 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_elasticsearch", 2024-03-11T18:44:16.090 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "elasticsearch container image", 2024-03-11T18:44:16.091 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_grafana": { 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_grafana", 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.092 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.093 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_haproxy": { 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_haproxy", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.094 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "HAproxy container image", 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_agent": { 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_agent", 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.095 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger agent container image", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.096 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_collector": { 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_collector", 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-11T18:44:16.097 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger collector container image", 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_jaeger_query": { 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_jaeger_query", 2024-03-11T18:44:16.098 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Jaeger query container image", 2024-03-11T18:44:16.099 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_keepalived": { 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_keepalived", 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.100 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Keepalived container image", 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.101 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_loki": { 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_loki", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.102 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Loki container image", 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_node_exporter": { 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_node_exporter", 2024-03-11T18:44:16.103 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.104 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_nvmeof": { 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_nvmeof", 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.105 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Nvme-of container image", 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.106 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_prometheus": { 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_prometheus", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.107 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Prometheus container image", 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_promtail": { 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_promtail", 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.108 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Promtail container image", 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.109 INFO:teuthology.orchestra.run.smithi002.stdout: "container_image_snmp_gateway": { 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_image_snmp_gateway", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.110 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "SNMP Gateway container image", 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "container_init": { 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "container_init", 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.111 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Run podman/docker with `--init`", 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "daemon_cache_timeout": { 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "daemon_cache_timeout", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "default_cephadm_command_timeout": { 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_cephadm_command_timeout", 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.114 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "900", 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "default_registry": { 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default_registry", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "docker.io", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.116 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "device_cache_timeout": { 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_cache_timeout", 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.117 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1800", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache device inventory", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.118 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "device_enhanced_scan": { 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_enhanced_scan", 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.119 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "facts_cache_timeout": { 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "facts_cache_timeout", 2024-03-11T18:44:16.120 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "seconds to cache host facts data", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.121 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "host_check_interval": { 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "host_check_interval", 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.122 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to perform a host check", 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "hw_monitoring": { 2024-03-11T18:44:16.123 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hw_monitoring", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.124 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "inventory_list_all": { 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "inventory_list_all", 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.125 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.126 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.127 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.128 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "log_refresh_metadata": { 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_refresh_metadata", 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.129 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.130 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-11T18:44:16.131 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.132 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.133 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.134 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.135 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.136 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-11T18:44:16.137 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "*", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.138 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "max_count_per_host": { 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_count_per_host", 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.139 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of daemons per service per host", 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "max_osd_draining_count": { 2024-03-11T18:44:16.140 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_osd_draining_count", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-11T18:44:16.141 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "migration_current": { 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "migration_current", 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.142 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "internal - do not modify", 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.143 INFO:teuthology.orchestra.run.smithi002.stdout: "mode": { 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mode", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "root", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.144 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm-package", 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "root" 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "mode for remote execution of cephadm", 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "oob_default_addr": { 2024-03-11T18:44:16.145 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "oob_default_addr", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "169.254.1.1", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-11T18:44:16.146 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "prometheus_alerts_path": { 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus_alerts_path", 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.147 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.148 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_insecure": { 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_insecure", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.149 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_password": { 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_password", 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.150 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.151 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_url": { 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_url", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "registry_username": { 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "registry_username", 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.153 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.154 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "secure_monitoring_stack": { 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "secure_monitoring_stack", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.155 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "service_discovery_port": { 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "service_discovery_port", 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.156 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8765", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "cephadm service discovery port", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.157 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "ssh_config_file": { 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssh_config_file", 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.158 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "use_agent": { 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_agent", 2024-03-11T18:44:16.159 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.160 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "use_repo_digest": { 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "use_repo_digest", 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.162 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_failed_host_check": { 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_failed_host_check", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.163 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if the host check fails", 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_daemons": { 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_daemons", 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.164 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.165 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_on_stray_hosts": { 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_on_stray_hosts", 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.166 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crash", 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.168 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.169 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.170 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.171 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.172 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.173 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.174 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.175 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "retain_interval": { 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retain_interval", 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.176 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "31536000", 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.177 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_recent_interval": { 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_recent_interval", 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1209600", 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.178 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.179 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dashboard", 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.180 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.181 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_HOST": { 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.182 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-11T18:44:16.183 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.184 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_ENABLED": { 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_ENABLED", 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.185 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-11T18:44:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.187 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "ENABLE_BROWSABLE_API": { 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.188 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.189 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.190 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.191 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.192 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.193 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.194 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.195 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.196 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.197 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.198 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.199 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-11T18:44:16.200 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.201 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_PASSWORD": { 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.202 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-11T18:44:16.203 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.204 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_URL": { 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_URL", 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_API_USERNAME": { 2024-03-11T18:44:16.206 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.207 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.208 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-11T18:44:16.209 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.210 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.211 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-11T18:44:16.212 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.213 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_HOST": { 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.214 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-11T18:44:16.215 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.216 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.217 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-11T18:44:16.218 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.219 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.221 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.222 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.223 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.224 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.225 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.226 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.227 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_ENABLED": { 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.228 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-11T18:44:16.229 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.230 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.231 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-11T18:44:16.232 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "45", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.235 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ACCESS_KEY": { 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.236 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.237 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "admin", 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.238 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SECRET_KEY": { 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.239 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "RGW_API_SSL_VERIFY": { 2024-03-11T18:44:16.240 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.241 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.242 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.243 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.244 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.245 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.246 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "cross_origin_url": { 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cross_origin_url", 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.247 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "crt_file": { 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "crt_file", 2024-03-11T18:44:16.248 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.249 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "debug": { 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "debug", 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.250 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Enable/disable debug options", 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "jwt_token_ttl": { 2024-03-11T18:44:16.251 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "jwt_token_ttl", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "28800", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.252 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.253 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.254 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.255 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.256 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.257 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.258 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.259 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.260 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.261 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "motd": { 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "motd", 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "The message of the day", 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_resolve_ip_addr": { 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "redirect_resolve_ip_addr", 2024-03-11T18:44:16.264 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.265 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.266 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:16.267 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8080", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.268 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.269 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.270 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl_server_port": { 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl_server_port", 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8443", 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.271 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:16.272 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "redirect", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.273 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect" 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:16.274 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.275 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "url_prefix": { 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url_prefix", 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.276 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.277 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_monitoring": { 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_monitoring", 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.278 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "monitor device health metrics", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.279 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.280 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.281 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.282 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.283 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.284 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.285 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.286 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.287 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "mark_out_threshold": { 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mark_out_threshold", 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "2419200", 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.288 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "pool_name": { 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pool_name", 2024-03-11T18:44:16.289 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "device_health_metrics", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-11T18:44:16.290 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "retention_period": { 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "retention_period", 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.291 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15552000", 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long to retain device health metrics", 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.292 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_frequency": { 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_frequency", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.293 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "self_heal": { 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "self_heal", 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.294 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.295 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.296 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how frequently to wake up and check device health", 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "warn_threshold": { 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "warn_threshold", 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.297 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7257600", 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.298 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "diskprediction_local", 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.299 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.300 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.301 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.302 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.303 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.304 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.305 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.306 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.307 INFO:teuthology.orchestra.run.smithi002.stdout: "predict_interval": { 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predict_interval", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "86400", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.308 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "predictor_model": { 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "predictor_model", 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.309 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "prophetstor", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.310 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "600", 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.311 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.312 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "influx", 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": false, 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "influxdb python module not found", 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "batch_size": { 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "batch_size", 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.313 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5000", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.314 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "database": { 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "database", 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "ceph", 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.315 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": { 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "hostname", 2024-03-11T18:44:16.316 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server hostname", 2024-03-11T18:44:16.317 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.318 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "30", 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "5", 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.319 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.320 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.321 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.322 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.323 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.324 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.325 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.326 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "password": { 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "password", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.328 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "password of InfluxDB server user", 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "port": { 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "port", 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.329 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "8086", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "InfluxDB server port", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.330 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "ssl": { 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ssl", 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "false", 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.331 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "threads": { 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threads", 2024-03-11T18:44:16.332 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "32", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.333 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "username": { 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "username", 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.334 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "username of InfluxDB server user", 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "verify_ssl": { 2024-03-11T18:44:16.335 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "verify_ssl", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "true", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-11T18:44:16.336 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "insights", 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.337 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.338 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.339 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.340 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.342 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.343 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.344 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.345 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "iostat", 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.346 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.347 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.348 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.349 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.350 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.351 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.352 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.353 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.354 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "k8sevents", 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_event_retention_days": { 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ceph_event_retention_days", 2024-03-11T18:44:16.355 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "7", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.356 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "config_check_secs": { 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "config_check_secs", 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.357 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "10", 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.358 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.359 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.360 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.361 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.363 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.364 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.365 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.366 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "localpool", 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "failure_domain": { 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "failure_domain", 2024-03-11T18:44:16.367 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "host", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "failure domain for any created local pool", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-11T18:44:16.368 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.369 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.370 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.371 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.372 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.373 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.374 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.375 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.376 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min_size": { 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "min_size", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.377 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default min_size for any created local pool", 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "num_rep": { 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "num_rep", 2024-03-11T18:44:16.378 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default replica count for any created local pool", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.379 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_num": { 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_num", 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "128", 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.380 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "default pg_num for any created local pool", 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "prefix": { 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prefix", 2024-03-11T18:44:16.381 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "name prefix for any created local pool", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.382 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "subtree": { 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subtree", 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "rack", 2024-03-11T18:44:16.383 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.384 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mds_autoscaler", 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.385 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.387 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.388 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.389 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.390 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.391 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.392 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirroring", 2024-03-11T18:44:16.393 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.394 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.395 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.396 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.397 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.398 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.399 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.400 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.401 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "nfs", 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.402 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.403 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.405 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.406 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.407 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.408 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.409 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.410 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "fail_fs": { 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "fail_fs", 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.411 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.412 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.413 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.414 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.415 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.416 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.417 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.418 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.419 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.420 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator": { 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "orchestrator", 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.421 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "cephadm", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "rook", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "test_orchestrator" 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Orchestrator backend", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.422 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_perf_query", 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.423 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.424 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.425 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.426 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.427 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.428 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.429 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.430 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.431 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "osd_support", 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.432 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.433 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.434 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.435 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.436 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.437 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.439 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "pg_autoscaler", 2024-03-11T18:44:16.440 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.442 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.443 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.444 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.445 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.446 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.447 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.448 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.449 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.450 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "threshold": { 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "threshold", 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.451 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "3.0", 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1.0", 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "scaling threshold", 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.452 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "progress", 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_pg_recovery_event": { 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_pg_recovery_event", 2024-03-11T18:44:16.453 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.454 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.455 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.456 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.457 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.458 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.459 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.459 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.459 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.459 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.459 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.460 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.460 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.460 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.460 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.460 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.461 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.462 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.463 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.464 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "max_completed_events": { 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_completed_events", 2024-03-11T18:44:16.465 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "50", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "number of past completed events to remember", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.466 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "sleep_interval": { 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sleep_interval", 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.467 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "5", 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "how long the module is going to sleep", 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "prometheus", 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "cache": { 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cache", 2024-03-11T18:44:16.469 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.470 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "exclude_perf_counters": { 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "exclude_perf_counters", 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.471 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.472 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.473 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.474 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.475 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.476 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.477 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.478 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.479 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.480 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools": { 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools", 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.481 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.482 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "300", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.483 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "scrape_interval": { 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "scrape_interval", 2024-03-11T18:44:16.484 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15.0", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.485 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.486 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "::", 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.487 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "9283", 2024-03-11T18:44:16.488 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.489 INFO:teuthology.orchestra.run.smithi002.stdout: "stale_cache_strategy": { 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stale_cache_strategy", 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "log", 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.490 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_behaviour": { 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_behaviour", 2024-03-11T18:44:16.491 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "default", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "default", 2024-03-11T18:44:16.492 INFO:teuthology.orchestra.run.smithi002.stdout: "error" 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "standby_error_status_code": { 2024-03-11T18:44:16.493 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "standby_error_status_code", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "500", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "400", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "599", 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.494 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2024-03-11T18:44:16.495 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.496 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.497 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.498 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.499 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.500 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.501 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.502 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.503 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_snap_create": { 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_snap_create", 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.504 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.505 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "mirror_snapshot_schedule": { 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "mirror_snapshot_schedule", 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.506 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "trash_purge_schedule": { 2024-03-11T18:44:16.507 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "trash_purge_schedule", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.508 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "restful", 2024-03-11T18:44:16.509 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "enable_auth": { 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enable_auth", 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.510 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.511 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "key_file": { 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "key_file", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.512 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.513 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.514 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.518 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.519 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.520 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.521 INFO:teuthology.orchestra.run.smithi002.stdout: "server_addr": { 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_addr", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.522 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "server_port": { 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "server_port", 2024-03-11T18:44:16.523 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.524 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rgw", 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.526 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.527 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.528 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.529 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.530 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.531 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.533 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.534 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rook", 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.535 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.536 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.538 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.539 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.540 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.541 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.542 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_class": { 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "storage_class", 2024-03-11T18:44:16.543 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "local", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.544 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "selftest", 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.545 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.546 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.547 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.548 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.549 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.550 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.551 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.552 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.553 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "roption1": { 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption1", 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.554 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "roption2": { 2024-03-11T18:44:16.555 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "roption2", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "xyz", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.556 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption1": { 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption1", 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.557 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.558 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption2": { 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption2", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.559 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption3": { 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption3", 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "float", 2024-03-11T18:44:16.560 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.561 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption4": { 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption4", 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.562 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.563 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption5": { 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption5", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.564 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption6": { 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption6", 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.565 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "rwoption7": { 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rwoption7", 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "1", 2024-03-11T18:44:16.567 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "42", 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "testkey": { 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testkey", 2024-03-11T18:44:16.568 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.569 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "testlkey": { 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testlkey", 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.570 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.571 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "testnewline": { 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "testnewline", 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.572 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "allow_m_granularity": { 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "allow_m_granularity", 2024-03-11T18:44:16.574 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "allow minute scheduled snapshots", 2024-03-11T18:44:16.575 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "dump_on_update": { 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "dump_on_update", 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.576 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "dump database to debug log on update", 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.578 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.579 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.580 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.581 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.582 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.583 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.584 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "stats", 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.587 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.588 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.589 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.590 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.591 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.592 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.593 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.594 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "status", 2024-03-11T18:44:16.595 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.596 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.597 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.598 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.600 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.601 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.602 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.603 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.604 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telegraf", 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "address": { 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "address", 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.605 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.606 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.607 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "15", 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.608 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.609 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.610 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.611 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.612 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.613 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.614 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.615 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.616 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "telemetry", 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.617 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_basic": { 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_basic", 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.618 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_crash": { 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_crash", 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.619 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.620 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_device": { 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_device", 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.621 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "True", 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.622 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_ident": { 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_ident", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.623 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "channel_perf": { 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "channel_perf", 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.624 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.625 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "contact": { 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "contact", 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.626 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "description": { 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "description", 2024-03-11T18:44:16.627 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.628 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "device_url": { 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "device_url", 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.629 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": { 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "enabled", 2024-03-11T18:44:16.630 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.631 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.632 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "24", 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "8", 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.633 INFO:teuthology.orchestra.run.smithi002.stdout: "last_opt_revision": { 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "last_opt_revision", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "1", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.634 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard": { 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard", 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.635 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.636 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "leaderboard_description": { 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "leaderboard_description", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.637 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.638 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.639 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.640 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.642 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "organization": { 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "organization", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "proxy": { 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "proxy", 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "url": { 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "url", 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-11T18:44:16.650 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.651 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "test_orchestrator", 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.652 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.653 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.654 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.655 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.656 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.657 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.658 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.659 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.660 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.661 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.662 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.663 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.664 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.665 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.666 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.667 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.668 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "max_concurrent_clones": { 2024-03-11T18:44:16.669 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "max_concurrent_clones", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "4", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.670 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "periodic_async_work": { 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "periodic_async_work", 2024-03-11T18:44:16.671 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Periodically check for async work", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.672 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot_clone_delay": { 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snapshot_clone_delay", 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.673 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "0", 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.674 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix", 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "can_run": true, 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "error_string": "", 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "module_options": { 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "discovery_interval": { 2024-03-11T18:44:16.675 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "discovery_interval", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "uint", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "100", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.676 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "identifier": { 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "identifier", 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.677 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "interval": { 2024-03-11T18:44:16.678 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "interval", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "secs", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "60", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.679 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "log_level": { 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_level", 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.680 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.681 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster": { 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster", 2024-03-11T18:44:16.682 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.683 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_cluster_level": { 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_cluster_level", 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.684 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "info", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [ 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "critical", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "debug", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "error", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "info", 2024-03-11T18:44:16.685 INFO:teuthology.orchestra.run.smithi002.stdout: "warning" 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "log_to_file": { 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "log_to_file", 2024-03-11T18:44:16.686 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "bool", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 1, 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "False", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.687 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_host": { 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_host", 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.688 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "", 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.689 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_port": { 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_port", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "int", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "10051", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.690 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "zabbix_sender": { 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "zabbix_sender", 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "str", 2024-03-11T18:44:16.691 INFO:teuthology.orchestra.run.smithi002.stdout: "level": "advanced", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "flags": 0, 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "min": "", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "max": "", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "enum_allowed": [], 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "desc": "", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "long_desc": "", 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-03-11T18:44:16.692 INFO:teuthology.orchestra.run.smithi002.stdout: "see_also": [] 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: "services": { 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: "dashboard": "https://172.21.15.2:8443/" 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.693 INFO:teuthology.orchestra.run.smithi002.stdout: "always_on_modules": { 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "octopus": [ 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2024-03-11T18:44:16.694 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "pacific": [ 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2024-03-11T18:44:16.695 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.696 INFO:teuthology.orchestra.run.smithi002.stdout: "quincy": [ 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2024-03-11T18:44:16.697 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "reef": [ 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2024-03-11T18:44:16.698 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "squid": [ 2024-03-11T18:44:16.699 INFO:teuthology.orchestra.run.smithi002.stdout: "balancer", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "crash", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "devicehealth", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "orchestrator", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "pg_autoscaler", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "progress", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "rbd_support", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "status", 2024-03-11T18:44:16.700 INFO:teuthology.orchestra.run.smithi002.stdout: "telemetry", 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: "volumes" 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: "last_failure_osd_epoch": 96, 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: "active_clients": [ 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "devicehealth", 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.701 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3581503461 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.702 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "libcephsqlite", 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2424865576 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.703 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "rbd_support", 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1099516780 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.704 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.705 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 406146731 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "snap_schedule", 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.706 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2087549707 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "volumes", 2024-03-11T18:44:16.707 INFO:teuthology.orchestra.run.smithi002.stdout: "addrvec": [ 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v2", 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "172.21.15.2:0", 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3820246949 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:16.708 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-03-11T18:44:16.709 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:44:16.710 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:16 smithi023 ceph-mon[36406]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:44:16.710 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:16 smithi023 ceph-mon[36406]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:44:16.711 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:16 smithi023 ceph-mon[36406]: pgmap v330: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:16.711 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:16 smithi165 ceph-mon[36634]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:44:16.711 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:16 smithi165 ceph-mon[36634]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:44:16.711 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:16 smithi165 ceph-mon[36634]: pgmap v330: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:16.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 msgr2=0x7fcb6c0a4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 0x7fcb6c0a4040 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcb78000f80 tx=0x7fcb78005f10 comp rx=0 tx=0).stop 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 msgr2=0x7fcb881995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000e970 tx=0x7fcb8000ed70 comp rx=0 tx=0).stop 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 shutdown_connections 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fcb6c0a1b80 0x7fcb6c0a4040 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb8810b580 0x7fcb881995e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:16.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb88102ad0 0x7fcb8819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:16.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 --2- 172.21.15.2:0/2646592217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb881021b0 0x7fcb8819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:16.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.574+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 >> 172.21.15.2:0/2646592217 conn(0x7fcb88076c10 msgr2=0x7fcb880fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:16.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.575+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 shutdown_connections 2024-03-11T18:44:16.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:14.575+0000 7fcb8da00640 1 -- 172.21.15.2:0/2646592217 wait complete. 2024-03-11T18:44:16.718 DEBUG:tasks.check_counter:Getting stats from a 2024-03-11T18:44:16.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-11T18:44:16.719 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.719 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T18:44:16.720 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='mgr.25418 ' entity='mgr.x' 2024-03-11T18:44:16.721 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:15 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2646592217' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-11T18:44:16.721 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:16 smithi002 ceph-mon[31328]: Adjusting osd_memory_target on smithi002 to 177.8M 2024-03-11T18:44:16.721 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:16 smithi002 ceph-mon[31328]: Unable to set osd_memory_target on smithi002 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-11T18:44:16.721 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:16 smithi002 ceph-mon[31328]: pgmap v330: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:17.196 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:44:18.458 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:18 smithi165 ceph-mon[36634]: pgmap v331: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:18.503 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:18 smithi002 ceph-mon[31328]: pgmap v331: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:18.656 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:18 smithi023 ceph-mon[36406]: pgmap v331: 129 pgs: 129 active+clean; 98 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 1.5 MiB/s wr, 179 op/s 2024-03-11T18:44:19.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:19 smithi002 ceph-mon[31328]: osdmap e108: 12 total, 12 up, 12 in 2024-03-11T18:44:19.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:19 smithi002 ceph-mon[31328]: pgmap v333: 129 pgs: 129 active+clean; 98 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 237, 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 225, 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1704016, 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 135392, 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:19.736 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.110397781, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.034861191, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.089107201, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.010684203, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 224, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.021831795, 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000097463 2024-03-11T18:44:19.737 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1704016, 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 135392 2024-03-11T18:44:19.738 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 172, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 185072, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.039723825, 2024-03-11T18:44:19.739 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.029081127, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.004868555, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 169, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.045563377, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000269605 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.740 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 185072 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2024-03-11T18:44:19.741 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 3024, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001543849, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000089623, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001439965, 2024-03-11T18:44:19.742 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001633148, 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000544382 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:19.743 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.744 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:19.745 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:19.746 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:19.747 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:19.748 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149785, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 32, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:19.749 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2024-03-11T18:44:19.750 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2024-03-11T18:44:19.751 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 239, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1670614, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 239, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 239, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1670614, 2024-03-11T18:44:19.752 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:19.753 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:44:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:19 smithi165 ceph-mon[36634]: osdmap e108: 12 total, 12 up, 12 in 2024-03-11T18:44:19.906 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:19 smithi165 ceph-mon[36634]: pgmap v333: 129 pgs: 129 active+clean; 98 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-03-11T18:44:19.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:19 smithi023 ceph-mon[36406]: osdmap e108: 12 total, 12 up, 12 in 2024-03-11T18:44:19.906 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:19 smithi023 ceph-mon[36406]: pgmap v333: 129 pgs: 129 active+clean; 98 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.2 MiB/s wr, 101 op/s 2024-03-11T18:44:20.830 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-11T18:44:20.831 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-11T18:44:20.831 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-11T18:44:20.831 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-11T18:44:20.831 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-11T18:44:20.831 DEBUG:tasks.check_counter:Getting stats from d 2024-03-11T18:44:20.832 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-11T18:44:21.277 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:44:22.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:21 smithi002 ceph-mon[31328]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:22.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:21 smithi165 ceph-mon[36634]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:22.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:21 smithi023 ceph-mon[36406]: pgmap v334: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:23.570 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:44:23.570 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.004141227, 2024-03-11T18:44:23.571 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000807099, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:23.572 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 178, 2024-03-11T18:44:23.573 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 186128, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.039701197, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.028305046, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.004941135, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:23.574 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 175, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.040111272, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000229207 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T18:44:23.575 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 186128 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 247, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 233, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1708672, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 139584, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:23.576 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.126347457, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.025802978, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.133973050, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.014228293, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 233, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.032568276, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000139778 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:23.577 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1708672, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 139584 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2024-03-11T18:44:23.578 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:23.579 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:23.580 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:23.581 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:23.582 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148714, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:23.583 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:23.584 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2024-03-11T18:44:23.585 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 245, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1671394, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 245, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 245, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1671394, 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2024-03-11T18:44:23.586 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:23.587 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:23.587 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:23.587 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:23.587 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:23.587 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:44:23.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:23 smithi002 ceph-mon[31328]: pgmap v335: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:24.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:23 smithi165 ceph-mon[36634]: pgmap v335: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:24.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:23 smithi023 ceph-mon[36406]: pgmap v335: 129 pgs: 129 active+clean; 96 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 1.0 MiB/s wr, 73 op/s 2024-03-11T18:44:24.380 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-11T18:44:24.381 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-11T18:44:24.381 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-11T18:44:24.381 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-11T18:44:24.381 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-11T18:44:24.381 DEBUG:tasks.check_counter:Getting stats from g 2024-03-11T18:44:24.381 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-11T18:44:24.897 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:44:26.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:25 smithi002 ceph-mon[31328]: pgmap v336: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:26.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:25 smithi165 ceph-mon[36634]: pgmap v336: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:26.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:25 smithi023 ceph-mon[36406]: pgmap v336: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:27.501 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:44:27.501 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:27.501 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 240, 2024-03-11T18:44:27.501 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 228, 2024-03-11T18:44:27.501 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1704832, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 137264, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.117766212, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.039210170, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.086093182, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.012013307, 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:27.502 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 227, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.025094179, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000110547 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1704832, 2024-03-11T18:44:27.503 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 137264 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 174, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 185424, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:27.504 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.039876653, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.028819284, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.004462333, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 171, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.033011203, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000193047 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.505 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 185424 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:27.506 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 4, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 4, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 3024, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 448, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001938123, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000084361, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.001773092, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:27.507 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 3, 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.001650067, 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000550022 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:27.508 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2024-03-11T18:44:27.509 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:27.510 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:27.511 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:27.512 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 148714, 2024-03-11T18:44:27.513 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 30, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:27.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2024-03-11T18:44:27.515 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 242, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1671004, 2024-03-11T18:44:27.516 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 242, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 242, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1671004, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:27.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:27.518 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:27.518 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:44:28.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:27 smithi002 ceph-mon[31328]: pgmap v337: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:28.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:27 smithi023 ceph-mon[36406]: pgmap v337: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:28.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:27 smithi165 ceph-mon[36634]: pgmap v337: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1002 KiB/s wr, 69 op/s 2024-03-11T18:44:28.305 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-11T18:44:28.305 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-11T18:44:28.305 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-11T18:44:28.305 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-11T18:44:28.305 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-11T18:44:28.305 DEBUG:tasks.check_counter:Getting stats from j 2024-03-11T18:44:28.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-11T18:44:28.781 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T18:44:30.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:29 smithi002 ceph-mon[31328]: pgmap v338: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 68 op/s 2024-03-11T18:44:30.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:29 smithi165 ceph-mon[36634]: pgmap v338: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 68 op/s 2024-03-11T18:44:30.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:29 smithi023 ceph-mon[36406]: pgmap v338: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.2 MiB/s wr, 68 op/s 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 250, 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 236, 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 1709488, 2024-03-11T18:44:30.995 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 141456, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.113160874, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.037328231, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.083395858, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.012759171, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 236, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.023961029, 2024-03-11T18:44:30.996 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000101529 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 1709488, 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 141456 2024-03-11T18:44:30.997 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 2, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 180, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 320, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 186480, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:30.998 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.038766915, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.027360005, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.005093717, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 177, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.036036962, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000203598 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:30.999 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 186480 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:31.000 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_messages": 0, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages": 2, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_bytes": 0, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_bytes": 0, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_total_time": 0.001145538, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_recv_time": 0.000975788, 2024-03-11T18:44:31.001 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:31.002 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "cct": { 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "total_workers": 0, 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "unhealthy_workers": 0 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.003 INFO:teuthology.orchestra.run.smithi002.stdout: "mempool": { 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:31.004 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:31.005 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:31.006 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_items": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:31.007 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_bytes": 149745, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_anon_items": 31, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_bytes": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_items": 0, 2024-03-11T18:44:31.008 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_items": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:31.009 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap_items": 0, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_co_items": 2, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_1_items": 0, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "unittest_2_items": 0 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-03-11T18:44:31.010 INFO:teuthology.orchestra.run.smithi002.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 0, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 104857600, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "get_started": 0, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "get": 248, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "get_sum": 1671784, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "get_or_fail_success": 248, 2024-03-11T18:44:31.011 INFO:teuthology.orchestra.run.smithi002.stdout: "take": 0, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "take_sum": 0, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "put": 248, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "put_sum": 1671784, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "wait": { 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "avgcount": 0, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "sum": 0.000000000, 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: "avgtime": 0.000000000 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:31.012 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-03-11T18:44:31.013 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-03-11T18:44:31.797 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-11T18:44:31.797 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-11T18:44:31.797 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-11T18:44:31.797 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-11T18:44:31.798 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-11T18:44:31.798 DEBUG:tasks.check_counter:Getting stats from b 2024-03-11T18:44:31.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-11T18:44:32.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:31 smithi002 ceph-mon[31328]: pgmap v339: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-11T18:44:32.064 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:44:32.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:31 smithi165 ceph-mon[36634]: pgmap v339: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-11T18:44:32.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:31 smithi023 ceph-mon[36406]: pgmap v339: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.0 MiB/s wr, 63 op/s 2024-03-11T18:44:33.894 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:33 smithi023 ceph-mon[36406]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1019 KiB/s wr, 52 op/s 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 92920, 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 130328, 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 22010495, 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 221472250, 2024-03-11T18:44:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 10, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 7.681642799, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 3.398703989, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 3.264902913, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 1.086110689, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 130327, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 14.215211974, 2024-03-11T18:44:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000109073 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 148038, 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.025987439, 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000175 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1785776, 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 471856 2024-03-11T18:44:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 253317, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 388488, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 84918991, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 431455438, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 100, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 6, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 16.148272117, 2024-03-11T18:44:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 6.682898650, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 9.756263874, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 1.283839882, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 388485, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 29.205262316, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000075177 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 426608, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.079232925, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000185 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 436528, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 362304 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 78354, 2024-03-11T18:44:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 115590, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 17305605, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 149670016, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 21, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 6, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 6.087448460, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 2.598665878, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 2.207183466, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 1.097452585, 2024-03-11T18:44:34.035 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 115588, 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 27.193927902, 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000235266 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 105146, 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.019136946, 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000182 2024-03-11T18:44:34.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 79761, 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 56.698406938, 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000710853 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 44454, 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 4.268230807, 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000096014 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 130629, 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 128995, 2024-03-11T18:44:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 128995, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 282.148760538, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.002187284 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 1634, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 1643, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 9, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 15, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 1272, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 353, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 909, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 10, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 769, 2024-03-11T18:44:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 15, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 98, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 98, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 3, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 162157, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 120011, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 1634, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 371, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 1, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 11, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 67, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 21137952, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 6, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 1, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 43801, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 104915, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 69326, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 14864, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 9293, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 80888 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 4, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-11T18:44:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 15, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 47, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 24488, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 24441, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 1, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-11T18:44:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 24, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 109034, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 107508, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 107508, 2024-03-11T18:44:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 1526, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 1945, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 180, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 178, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 178, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 2, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 1, 2024-03-11T18:44:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 3, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 383535692, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 388201398, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 0, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 39244, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 171.973582493, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.004382162 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 0 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 636, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 25001, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 24365, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 37, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 119, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 82, 2024-03-11T18:44:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 1272, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 45422, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 44150, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 98, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 24797, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 24699, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 70516, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 215420 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 192830, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 3722, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 130629, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 2345, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 7444, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 24555, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 24.023663723, 2024-03-11T18:44:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000978361 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3848, 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 30.764286308, 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.007994876 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 22345, 2024-03-11T18:44:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 16.548217488, 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000740578 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-11T18:44:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000145167, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000145167 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-11T18:44:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 11, 2024-03-11T18:44:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.003163050, 2024-03-11T18:44:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000287550 2024-03-11T18:44:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 20, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.023351654, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001167582 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 11, 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1.104191986, 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.100381089 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 48018, 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 170.728136980, 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.003555502 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-11T18:44:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4808, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 5.652604926, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001175666 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 5, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.042511143, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.008502228 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4, 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.016171658, 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.004042914 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 872, 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.290850171, 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000333543 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-11T18:44:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 9, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.024494651, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.002721627 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 24488, 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 32.926971633, 2024-03-11T18:44:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001344616 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 2, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 6, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 4, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 2, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 20042, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 10021, 2024-03-11T18:44:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 647, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 120, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 120, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 1601806, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 824, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 176, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 2, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-11T18:44:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 206, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 2802222, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 49806, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-11T18:44:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 118058, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 406890042, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 31, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 118058, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 118058, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 3262.821885901, 2024-03-11T18:44:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.027637448 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 118058, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 160846 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 118058, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 538, 2024-03-11T18:44:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 117520, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1525, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 25284, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 538, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 39782, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 490, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-11T18:44:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 9, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 49646, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 38138, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-11T18:44:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-11T18:44:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 108, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 22, 2024-03-11T18:44:34.086 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 12, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 12, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 4210, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 0, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 1224 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-11T18:44:34.087 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 1, 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-11T18:44:34.088 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 16, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 8, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 48882, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 24441, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:34.089 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 424587, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 85957581, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 424587, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:34.090 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 424587, 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 85957581, 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:34.091 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 118058, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 409556807, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 118058, 2024-03-11T18:44:34.092 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 409556807, 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:34.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 118058, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 118058, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 118058, 2024-03-11T18:44:34.094 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 118058, 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:34.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 24441, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 2666765, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 24441, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 538, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2666765, 2024-03-11T18:44:34.096 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x5558acddc640": { 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 51514, 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:34.097 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 109034, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 384007094, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 109034, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 39244, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 383955580, 2024-03-11T18:44:34.098 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:34.099 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T18:44:34.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:33 smithi165 ceph-mon[36634]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1019 KiB/s wr, 52 op/s 2024-03-11T18:44:34.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:33 smithi002 ceph-mon[31328]: pgmap v340: 129 pgs: 129 active+clean; 96 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 1019 KiB/s wr, 52 op/s 2024-03-11T18:44:35.035 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-03-11T18:44:35.035 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-11T18:44:35.035 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-11T18:44:35.035 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-11T18:44:35.035 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-11T18:44:35.035 DEBUG:tasks.check_counter:Getting stats from e 2024-03-11T18:44:35.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-11T18:44:35.298 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:44:36.057 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:35 smithi002 ceph-mon[31328]: pgmap v341: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-11T18:44:36.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:35 smithi165 ceph-mon[36634]: pgmap v341: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-11T18:44:36.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:35 smithi023 ceph-mon[36406]: pgmap v341: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-11T18:44:37.103 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T18:44:37.103 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:37.103 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 790, 2024-03-11T18:44:37.103 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 792, 2024-03-11T18:44:37.103 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 12785763, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 245754, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 7, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.067587654, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.020919411, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.022583432, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.017631003, 2024-03-11T18:44:37.104 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 790, 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.024332789, 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000030800 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 805, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 981, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 12786730, 2024-03-11T18:44:37.106 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 788583, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 9, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 6, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.077871797, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.021671274, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.028482076, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.020653650, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 978, 2024-03-11T18:44:37.107 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.063321693, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000064746 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-03-11T18:44:37.108 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 540400 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1445, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 1426, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 8587702, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 508702, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:37.109 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.146560896, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.042629609, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.069349182, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.037622091, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1425, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.050697784, 2024-03-11T18:44:37.110 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000035577 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1760496, 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 144832 2024-03-11T18:44:37.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4764, 2024-03-11T18:44:37.112 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.107514692, 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000022568 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.113 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.114 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-11T18:44:37.115 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 2699, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 2699, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 2699, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-11T18:44:37.116 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 3, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-11T18:44:37.117 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:37.118 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:37.119 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:37.120 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-11T18:44:37.121 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:37.122 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-11T18:44:37.123 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 20282, 2024-03-11T18:44:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 24, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 3, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-11T18:44:37.125 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 35821572, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 35816638, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 20282 2024-03-11T18:44:37.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 85, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 2698, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 2613, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 22, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 22, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 0, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 2699, 2024-03-11T18:44:37.127 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 2699, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 52236, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 215420 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-11T18:44:37.128 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.130 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.131 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.132 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.134 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.135 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.136 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.136 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-11T18:44:37.136 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.136 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-11T18:44:37.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.139 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.140 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.142 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.143 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.144 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-11T18:44:37.145 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.146 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 2, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 5, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 3, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 3, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-11T18:44:37.147 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 2982, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:37.148 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:37.149 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:37.150 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:37.151 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 217830, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 138, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:37.152 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 206, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:37.153 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 2170632, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 14062, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.154 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 2758, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 2758, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2758, 2024-03-11T18:44:37.155 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2.661974600, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000965182 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2758, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2759 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 2758, 2024-03-11T18:44:37.156 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 2758, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1537, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 1219, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-11T18:44:37.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-11T18:44:37.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:37.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-11T18:44:37.160 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-11T18:44:37.161 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 108, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 17, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 8, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 8, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-11T18:44:37.162 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.163 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 0, 2024-03-11T18:44:37.164 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 3036, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 33869860, 2024-03-11T18:44:37.165 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:37.166 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 3036, 2024-03-11T18:44:37.166 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:37.166 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:37.166 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 3036, 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 33869860, 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.168 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:37.169 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2758, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 31629863, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2758, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 31629863, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:37.171 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2758, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 2758, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2758, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2758, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.172 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:37.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.174 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x55ffc10ca960": { 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:37.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:37.176 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:37.177 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:37.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:37.177 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T18:44:37.954 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-11T18:44:37.954 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-11T18:44:37.954 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-11T18:44:37.955 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-11T18:44:37.955 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T18:44:37.955 DEBUG:tasks.check_counter:Getting stats from h 2024-03-11T18:44:37.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-11T18:44:38.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:37 smithi165 ceph-mon[36634]: pgmap v342: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-11T18:44:38.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:37 smithi023 ceph-mon[36406]: pgmap v342: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-11T18:44:38.204 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:44:38.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:37 smithi002 ceph-mon[31328]: pgmap v342: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-03-11T18:44:39.977 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:39 smithi023 ceph-mon[36406]: pgmap v343: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.2 MiB/s wr, 60 op/s 2024-03-11T18:44:39.977 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T18:44:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:39.977 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 99653, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 62346, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 46260294, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 27481956, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 10, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 4, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 4.759864693, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 2.035138097, 2024-03-11T18:44:39.978 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 3.385459601, 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.183944746, 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 62346, 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1.101171627, 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000017662 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:39.979 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 148918, 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.018112354, 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000121 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:39.980 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 35504, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 48106, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 12999558, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 43377426, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 7, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 6, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 2.331887581, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.898953976, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 1.565789763, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.165944077, 2024-03-11T18:44:39.981 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 48106, 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1.012228413, 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000021041 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 63471, 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.010601011, 2024-03-11T18:44:39.982 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000167 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1771984, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 148496 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 3245, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 3457, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1092233, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 6008064, 2024-03-11T18:44:39.983 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 17, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 7, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.235417887, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.083320270, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.060086912, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.075125816, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3452, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.141797791, 2024-03-11T18:44:39.984 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000041076 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 84, 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000014617, 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000174 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 7488, 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 480848 2024-03-11T18:44:39.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 17807, 2024-03-11T18:44:39.986 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 6.386839352, 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000358670 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 5478, 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.356460867, 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000065071 2024-03-11T18:44:39.987 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 18060, 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 14745, 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 14745, 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 17.651590000, 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001197123 2024-03-11T18:44:39.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 3315, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 11, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 92, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 2, 2024-03-11T18:44:39.989 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 90, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 86, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 13, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 14, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 3, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 38451, 2024-03-11T18:44:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 18641, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 3312, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 5, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 873, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-11T18:44:39.991 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 1, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 11, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 68, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 21133856, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 6, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:39.992 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 96, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 10068, 2024-03-11T18:44:39.993 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 8077, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 6632, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 949, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 14911 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-11T18:44:39.994 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 6, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 4, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 8, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 4, 2024-03-11T18:44:39.997 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 2625, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 2621, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:39.998 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:39.999 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 20302, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 20302, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 24, 2024-03-11T18:44:40.000 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 24, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 3, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 35887543, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 0, 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-11T18:44:40.001 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 8695, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 30.149604230, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.003467464 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 0 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 86, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 2703, 2024-03-11T18:44:40.002 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 2617, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 31, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 33, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 2, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 91, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 2926, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 2835, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 5, 2024-03-11T18:44:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 2705, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 2700, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 60024, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 207228 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 30761, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 18060, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 1337, 2024-03-11T18:44:40.004 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 3722, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2625, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 5.554776251, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.002116105 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 70, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.049017561, 2024-03-11T18:44:40.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000700250 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4134, 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 4.423926922, 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001070132 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.007 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1, 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000126802, 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000126802 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-11T18:44:40.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-11T18:44:40.011 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.012 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.013 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4415, 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2.187933725, 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000495568 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 873, 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1.966852569, 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.002252981 2024-03-11T18:44:40.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.015 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.016 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2, 2024-03-11T18:44:40.017 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007769570, 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.003884785 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:40.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2625, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 3.461186600, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001318547 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:40.021 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 2, 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 5, 2024-03-11T18:44:40.022 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 3, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 2, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 1847, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 923, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 644, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:40.023 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:40.024 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:40.025 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:40.026 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:40.027 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 383716, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 189, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:40.028 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 206, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 368704, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 4387, 2024-03-11T18:44:40.029 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 15566, 2024-03-11T18:44:40.030 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 32265845, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 27, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 15566, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 15566, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 83.331204308, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.005353411 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-11T18:44:40.031 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 15566, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 16217 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 15566, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 442, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 15124, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-11T18:44:40.032 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 0, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 10, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 442, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 9137, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 271, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 5261, 2024-03-11T18:44:40.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 20, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-11T18:44:40.034 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-11T18:44:40.035 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-11T18:44:40.036 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 108, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 17, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 12, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 12, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-11T18:44:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 828, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 0, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 248 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 1, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-11T18:44:40.038 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 16, 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-11T18:44:40.039 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 8, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 5242, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 2621, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:40.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 138398, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 47876806, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 138398, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 138398, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 47876806, 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:40.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:40.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 15566, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 32551822, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 15566, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 32551822, 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:40.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:40.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 15566, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 15566, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 15566, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 15566, 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:40.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:40.046 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 2621, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 285977, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 2621, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 442, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 285977, 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:40.047 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x5626fc5f4640": { 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 32648, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:40.048 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 20302, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 31693239, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 20302, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 8695, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 31660591, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:40.049 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.050 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:40.050 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T18:44:40.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:39 smithi165 ceph-mon[36634]: pgmap v343: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.2 MiB/s wr, 60 op/s 2024-03-11T18:44:40.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:39 smithi002 ceph-mon[31328]: pgmap v343: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 1.2 MiB/s wr, 60 op/s 2024-03-11T18:44:40.883 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-11T18:44:40.883 INFO:tasks.check_counter:Daemon mds.h mds.imported=1 2024-03-11T18:44:40.883 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-11T18:44:40.883 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-11T18:44:40.883 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T18:44:40.883 DEBUG:tasks.check_counter:Getting stats from k 2024-03-11T18:44:40.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-11T18:44:41.135 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T18:44:42.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:41 smithi165 ceph-mon[36634]: pgmap v344: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 843 KiB/s wr, 48 op/s 2024-03-11T18:44:42.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:41 smithi023 ceph-mon[36406]: pgmap v344: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 843 KiB/s wr, 48 op/s 2024-03-11T18:44:42.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:41 smithi002 ceph-mon[31328]: pgmap v344: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 843 KiB/s wr, 48 op/s 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-11T18:44:42.853 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001257750, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000300345, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:42.854 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.855 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 6, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 184, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 3344, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 187280, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:42.856 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.010606096, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.006651100, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.002043804, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 182, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.038137129, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000209544 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.857 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 187280 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:42.858 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 250, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 238, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1707552, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 143504, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.044355160, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.012325527, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.036014564, 2024-03-11T18:44:42.859 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.005492086, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 237, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008440001, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000035611 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:42.860 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1707552, 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 143504 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148748, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-11T18:44:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-11T18:44:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 252, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1672304, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 252, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 252, 2024-03-11T18:44:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1672304, 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-11T18:44:42.870 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-11T18:44:43.764 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-11T18:44:43.764 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-11T18:44:43.765 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-11T18:44:43.765 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-11T18:44:43.765 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-11T18:44:43.765 DEBUG:tasks.check_counter:Getting stats from c 2024-03-11T18:44:43.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-11T18:44:44.021 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:44:44.116 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:43 smithi023 ceph-mon[36406]: pgmap v345: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 685 KiB/s wr, 38 op/s 2024-03-11T18:44:44.142 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:43 smithi165 ceph-mon[36634]: pgmap v345: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 685 KiB/s wr, 38 op/s 2024-03-11T18:44:44.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:43 smithi002 ceph-mon[31328]: pgmap v345: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 685 KiB/s wr, 38 op/s 2024-03-11T18:44:45.759 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 412, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 414, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 32085609, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 128142, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 7, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:45.760 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.053144674, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.010380609, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.027375787, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.011501893, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 412, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.017108643, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000041525 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.761 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.762 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 247, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 246, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 42142425, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 89671, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.037146122, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.006427741, 2024-03-11T18:44:45.763 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.025302186, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.004852914, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 245, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.012527497, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000051132 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.764 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 174640, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 33936 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 228, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 266, 2024-03-11T18:44:45.765 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 42003971, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 224234, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 6, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 5, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.038316905, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.005915897, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.025607257, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.005074643, 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:45.766 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 265, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.012315061, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000046471 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.767 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 154880 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:45.768 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1324, 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.068138108, 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000051463 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:45.769 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 0, 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 0, 2024-03-11T18:44:45.770 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 0, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:45.771 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 0, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 0, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 6471, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 4023, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 2448, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 0, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 821, 2024-03-11T18:44:45.772 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 3, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:45.773 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:45.774 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 67, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 21137952, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 6, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:45.775 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:45.776 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 0 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 0, 2024-03-11T18:44:45.777 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:45.778 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-03-11T18:44:45.779 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 0, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 0, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 0, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 0, 2024-03-11T18:44:45.780 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 26192, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 0, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 3, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 12, 2024-03-11T18:44:45.781 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 0, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 383535692, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 388477224, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 388475136, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.782 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 26192 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 227, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 6334, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 6107, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 37, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 44, 2024-03-11T18:44:45.783 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 7, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 6471, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 6471, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 0, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 0, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 0, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 0, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 56804, 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 215396 2024-03-11T18:44:45.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.785 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:45.786 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.787 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.789 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.790 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-03-11T18:44:45.791 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.792 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.792 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.792 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.793 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.794 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.795 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.796 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.798 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:45.799 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-03-11T18:44:45.800 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.801 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:45.802 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 2, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 2, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 0, 2024-03-11T18:44:45.803 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 0, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 0, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 0, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 2266, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:45.804 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:45.805 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:45.806 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:45.807 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 233875, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 398, 2024-03-11T18:44:45.808 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 206, 2024-03-11T18:44:45.809 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 5313982, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 38798, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-03-11T18:44:45.810 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 814, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 26, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 4, 2024-03-11T18:44:45.811 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 814, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 814, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1.235979150, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.001518401 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 814, 2024-03-11T18:44:45.812 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 815 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 814, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 814, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 0, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 438, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 0, 2024-03-11T18:44:45.813 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 373, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-03-11T18:44:45.814 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 2, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-03-11T18:44:45.815 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-03-11T18:44:45.816 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-03-11T18:44:45.817 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 108, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 6, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 9, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 9, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-03-11T18:44:45.818 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 0, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 2, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 0 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 0, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-03-11T18:44:45.819 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 0 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-03-11T18:44:45.820 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 0, 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 0, 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 0, 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:45.821 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 883, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 116146060, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 883, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:45.822 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 883, 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 116146060, 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.823 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 814, 2024-03-11T18:44:45.824 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 115824528, 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 814, 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 115824528, 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:45.826 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 814, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 814, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 814, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 814, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.827 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:45.828 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.829 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x55a999038960": { 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:45.830 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:45.831 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:45.832 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:45.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:45.832 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T18:44:46.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:45 smithi023 ceph-mon[36406]: pgmap v346: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 765 KiB/s wr, 46 op/s 2024-03-11T18:44:46.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:45 smithi165 ceph-mon[36634]: pgmap v346: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 765 KiB/s wr, 46 op/s 2024-03-11T18:44:46.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:45 smithi002 ceph-mon[31328]: pgmap v346: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 765 KiB/s wr, 46 op/s 2024-03-11T18:44:46.592 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-11T18:44:46.592 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-11T18:44:46.592 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-11T18:44:46.592 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-11T18:44:46.592 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T18:44:46.593 DEBUG:tasks.check_counter:Getting stats from f 2024-03-11T18:44:46.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-11T18:44:46.857 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:44:48.042 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:47 smithi165 ceph-mon[36634]: pgmap v347: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 270 KiB/s wr, 19 op/s 2024-03-11T18:44:48.151 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:47 smithi002 ceph-mon[31328]: pgmap v347: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 270 KiB/s wr, 19 op/s 2024-03-11T18:44:48.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:47 smithi023 ceph-mon[36406]: pgmap v347: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 270 KiB/s wr, 19 op/s 2024-03-11T18:44:48.603 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 1103, 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 1084, 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 2038576, 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 402966, 2024-03-11T18:44:48.604 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 5, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.130758058, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.038533418, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.060449060, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.034361986, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1083, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.050514205, 2024-03-11T18:44:48.605 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000046642 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1761040, 2024-03-11T18:44:48.606 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 146080 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 8, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 186, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 3900, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 470172, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 6, 2024-03-11T18:44:48.607 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.010252441, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.006292847, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.002044936, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.000070132, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 184, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.028738045, 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000156185 2024-03-11T18:44:48.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 469488 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.609 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 1576, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 1577, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 404239, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 489832, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.093172951, 2024-03-11T18:44:48.610 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.030236000, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.022419636, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.034101920, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 1575, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.051672501, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000032807 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:48.611 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 0 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-03-11T18:44:48.612 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4053, 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.098650773, 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000024340 2024-03-11T18:44:48.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.614 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 0, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 0, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.615 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 0, 2024-03-11T18:44:48.616 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 1, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 0, 2024-03-11T18:44:48.617 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 0, 2024-03-11T18:44:48.618 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 0, 2024-03-11T18:44:48.619 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:48.620 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 0, 2024-03-11T18:44:48.621 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 0 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 0, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 0, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 0, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 0, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 0, 2024-03-11T18:44:48.622 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:48.623 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.624 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 9, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 0, 2024-03-11T18:44:48.625 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 0, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 0, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 2, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 2, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 0, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 4194304, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 4196798, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 4196657, 2024-03-11T18:44:48.626 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 9 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-03-11T18:44:48.627 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 3, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 3, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 0, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 1, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 1, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 0, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 0, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 0, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 0, 2024-03-11T18:44:48.628 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 0, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 0, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 0, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 47948, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 215420 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 0, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 0, 2024-03-11T18:44:48.629 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 0, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 0, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 0, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-03-11T18:44:48.630 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.632 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-03-11T18:44:48.633 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.634 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.635 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-03-11T18:44:48.636 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.637 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.637 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.637 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.638 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.639 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:48.640 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.642 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-03-11T18:44:48.643 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.644 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-03-11T18:44:48.646 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:48.647 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 1, 2024-03-11T18:44:48.648 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 4, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 3, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 2, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 0, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 0, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 3880, 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:48.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:48.650 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:48.651 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:48.652 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-03-11T18:44:48.653 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 182037, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 48, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-03-11T18:44:48.654 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 206, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:48.655 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 8320, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 13, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-03-11T18:44:48.656 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 2403, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 21, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 3, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 2403, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2403, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 1.475975211, 2024-03-11T18:44:48.657 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000614221 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2403, 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 2404 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 2403, 2024-03-11T18:44:48.658 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 2403, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 1565, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 836, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 0, 2024-03-11T18:44:48.659 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 1, 2024-03-11T18:44:48.660 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-03-11T18:44:48.661 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-03-11T18:44:48.662 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-03-11T18:44:48.663 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 108, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 17, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 4, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 4, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 0, 2024-03-11T18:44:48.664 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 2, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 0 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 0, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-03-11T18:44:48.665 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 0 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 0, 2024-03-11T18:44:48.666 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 0, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 0, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 2683, 2024-03-11T18:44:48.667 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 2188089, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 2683, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 2683, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 2188089, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.668 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:48.669 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 2403, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 2499, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 2403, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 2499, 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:48.670 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-03-11T18:44:48.671 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 2403, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 2403, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 2403, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 2403, 2024-03-11T18:44:48.672 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:48.673 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-03-11T18:44:48.674 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x55967df64960": { 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:48.675 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-03-11T18:44:48.676 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:48.677 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T18:44:49.418 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-11T18:44:49.418 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-11T18:44:49.418 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-11T18:44:49.418 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-11T18:44:49.418 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T18:44:49.418 DEBUG:tasks.check_counter:Getting stats from i 2024-03-11T18:44:49.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-11T18:44:49.660 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:44:50.045 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:49 smithi165 ceph-mon[36634]: pgmap v348: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 281 KiB/s wr, 23 op/s 2024-03-11T18:44:50.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:49 smithi023 ceph-mon[36406]: pgmap v348: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 281 KiB/s wr, 23 op/s 2024-03-11T18:44:50.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:49 smithi002 ceph-mon[31328]: pgmap v348: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 281 KiB/s wr, 23 op/s 2024-03-11T18:44:51.405 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T18:44:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:51.405 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 74755, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 37479, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 41369779, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 6288629, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 7, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 3.109002186, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 1.227861011, 2024-03-11T18:44:51.406 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 2.806444937, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.010411715, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 37478, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.572486395, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000015275 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 111425, 2024-03-11T18:44:51.407 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.013131834, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000117 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1774208, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 150256 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 33, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 215, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 8248, 2024-03-11T18:44:51.408 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 518348, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 15, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 4, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.015089370, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.007275328, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.005593923, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.000265343, 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:51.409 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 212, 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.043227818, 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000203904 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 38, 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000005856, 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000154 2024-03-11T18:44:51.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 4880, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 474208 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 3316, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 3332, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 993717, 2024-03-11T18:44:51.411 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 344002, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 7, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 5, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.261711335, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.094292399, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.224331653, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.000544578, 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:51.412 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 3331, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.069383028, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000020829 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 6601, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.001325983, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000200 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.413 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 1, 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-MDSRank": { 2024-03-11T18:44:51.414 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 68, 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.001594220, 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000023444 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "finisher-PurgeQueue": { 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "queue_len": 0, 2024-03-11T18:44:51.415 INFO:teuthology.orchestra.run.smithi165.stdout: "complete_latency": { 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2, 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000007236, 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000003618 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "mds": { 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "request": 3290, 2024-03-11T18:44:51.416 INFO:teuthology.orchestra.run.smithi165.stdout: "reply": 4, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "reply_latency": { 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 4, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.001271812, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000317953 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_reply": 0, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "forward": 3286, 2024-03-11T18:44:51.417 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_complete": 0, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_fetch_keys": 0, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_commit": 11, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_split": 0, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_merge": 0, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes": 21, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_top": 1, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_bottom": 20, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pin_tail": 0, 2024-03-11T18:44:51.418 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_pinned": 19, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_expired": 1, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "inodes_with_caps": 0, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "caps": 0, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "subtrees": 2, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse": 3299, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_hit": 8, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_forward": 3283, 2024-03-11T18:44:51.419 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_discover": 4, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_dir_fetch": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_remote_ino": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "traverse_lock": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "load_cent": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "q": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "exported": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "exported_inodes": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "imported": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "imported_inodes": 0, 2024-03-11T18:44:51.420 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_dir_fetch": 0, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_backtrace_fetch": 0, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "openino_peer_discover": 1, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rfiles": 68, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rbytes": 21133856, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "root_rsnaps": 6, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_fetch": 0, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_set_tag": 0, 2024-03-11T18:44:51.421 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_backtrace_repaired": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_inotable_repaired": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_inodes": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dir_base_inodes": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_dirfrag_rstats": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "scrub_file_inodes": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_inode_file_caps": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_revoke": 0, 2024-03-11T18:44:51.422 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_grant": 0, 2024-03-11T18:44:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_trunc": 0, 2024-03-11T18:44:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-11T18:44:51.423 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-11T18:44:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps": 0, 2024-03-11T18:44:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_caps_dirty": 0, 2024-03-11T18:44:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_cap_release": 0, 2024-03-11T18:44:51.426 INFO:teuthology.orchestra.run.smithi165.stdout: "process_request_cap_release": 2 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_cache": { 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update": 0, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_update_receipt": 6, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_try_discover": 4, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_send_discover": 7, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "dir_handle_discover": 0, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays": 0, 2024-03-11T18:44:51.427 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_delayed": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "num_strays_enqueuing": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_created": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_enqueued": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_reintegrated": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "strays_migrated": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_processing": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_enqueued": 0, 2024-03-11T18:44:51.428 INFO:teuthology.orchestra.run.smithi165.stdout: "num_recovering_prioritized": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_started": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "recovery_completed": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_enqueue_scrub": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_exportdir": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_flush": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragmentdir": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_fragstats": 0, 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: "ireq_inodestats": 0 2024-03-11T18:44:51.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_log": { 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evlrg": 0, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evadd": 9, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evex": 0, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evtrm": 0, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "ev": 9, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evexg": 0, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "evexd": 0, 2024-03-11T18:44:51.430 INFO:teuthology.orchestra.run.smithi165.stdout: "segadd": 2, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "segex": 0, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "segtrm": 0, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "seg": 2, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "segmjr": 2, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "segexg": 0, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "segexd": 0, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "expos": 4194304, 2024-03-11T18:44:51.431 INFO:teuthology.orchestra.run.smithi165.stdout: "wrpos": 4196798, 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "rdpos": 0, 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "jlat": { 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 8, 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.025792598, 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.003224074 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: "replayed": 0 2024-03-11T18:44:51.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_mem": { 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "ino": 24, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "ino+": 24, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "ino-": 0, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dir": 30, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dir+": 32, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dir-": 2, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dn": 21, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dn+": 22, 2024-03-11T18:44:51.433 INFO:teuthology.orchestra.run.smithi165.stdout: "dn-": 1, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "cap": 0, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "cap+": 3, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "cap-": 3, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "rss": 52524, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "heap": 207228 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_server": { 2024-03-11T18:44:51.434 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_client_request": 3292, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "dispatch_server_request": 0, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_request": 3290, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_client_session": 20, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "handle_peer_request": 0, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "req_create_latency": { 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getattr_latency": { 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2, 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000726705, 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000363352 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getfilelock_latency": { 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.436 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "req_link_latency": { 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookup_latency": { 2024-03-11T18:44:51.437 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 2, 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000545107, 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000272553 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookuphash_latency": { 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupino_latency": { 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupname_latency": { 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.439 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupparent_latency": { 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.440 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lookupsnap_latency": { 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "req_lssnap_latency": { 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.441 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mkdir_latency": { 2024-03-11T18:44:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.442 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mknod_latency": { 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.443 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "req_mksnap_latency": { 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "req_open_latency": { 2024-03-11T18:44:51.444 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "req_readdir_latency": { 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rename_latency": { 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "req_renamesnap_latency": { 2024-03-11T18:44:51.446 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "req_snapdiff_latency": { 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.447 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmdir_latency": { 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmsnap_latency": { 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "req_rmxattr_latency": { 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setattr_latency": { 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.449 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setdirlayout_latency": { 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setfilelock_latency": { 2024-03-11T18:44:51.450 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setlayout_latency": { 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "req_setxattr_latency": { 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "req_symlink_latency": { 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.452 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "req_unlink_latency": { 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_revoke_eviction": 0, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "cap_acquisition_throttle": 0, 2024-03-11T18:44:51.453 INFO:teuthology.orchestra.run.smithi165.stdout: "req_getvxattr_latency": { 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_sessions": { 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "session_count": 1, 2024-03-11T18:44:51.454 INFO:teuthology.orchestra.run.smithi165.stdout: "session_add": 4, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "session_remove": 3, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_open": 1, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "sessions_stale": 0, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "total_load": 3, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "average_load": 3, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "avg_session_uptime": 663, 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: "mdthresh_evicted": 0 2024-03-11T18:44:51.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:51.456 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:51.457 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:51.458 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:51.459 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 251332, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 107, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:51.460 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 9840, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 206, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:51.461 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 115551, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 589, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.462 INFO:teuthology.orchestra.run.smithi165.stdout: "objecter": { 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_active": 0, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_laggy": 0, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send": 48, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_send_bytes": 13121, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_resend": 33, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_reply": 48, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "op_latency": { 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 48, 2024-03-11T18:44:51.463 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.360993940, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.007520707 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "op_inflight": 0, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "oplen_avg": { 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 48, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 72 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "op": 48, 2024-03-11T18:44:51.464 INFO:teuthology.orchestra.run.smithi165.stdout: "op_r": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "op_w": 48, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "op_rmw": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "op_pg": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_stat": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_create": 10, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_read": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_write": 8, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writefull": 9, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_writesame": 0, 2024-03-11T18:44:51.465 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_append": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_zero": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_truncate": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_delete": 6, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_mapext": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_sparse_read": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_clonerange": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_getxattr": 0, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_setxattr": 20, 2024-03-11T18:44:51.466 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_cmpxattr": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_rmxattr": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_resetxattrs": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_call": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_watch": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_notify": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_src_cmpxattr": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_pgls_filter": 0, 2024-03-11T18:44:51.467 INFO:teuthology.orchestra.run.smithi165.stdout: "osdop_other": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_active": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_send": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_resend": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "linger_ping": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_active": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_send": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "poolop_resend": 0, 2024-03-11T18:44:51.468 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_active": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_send": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "poolstat_resend": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_active": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_send": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "statfs_resend": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "command_active": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "command_send": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "command_resend": 0, 2024-03-11T18:44:51.469 INFO:teuthology.orchestra.run.smithi165.stdout: "map_epoch": 108, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "map_full": 0, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "map_inc": 17, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_sessions": 9, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_open": 9, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_session_close": 0, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_laggy": 0, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_wr": 17, 2024-03-11T18:44:51.470 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_rd": 0, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_del": 2 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "oft": { 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_objs": 1, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_kv_pairs": 0, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_updates": 0, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "omap_total_removes": 0 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.471 INFO:teuthology.orchestra.run.smithi165.stdout: "osd-slow-ops": { 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "slow_ops_count": 0 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "purge_queue": { 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops": 0, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_ops_high_water": 0, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing": 0, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executing_high_water": 0, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed_ops": 0, 2024-03-11T18:44:51.472 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_executed": 0, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "pq_item_in_journal": 0 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 78100, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 35323358, 2024-03-11T18:44:51.473 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 78100, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 78100, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 35323358, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.474 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_bytes": { 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:51.475 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 48, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 13121, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 48, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 13121, 2024-03-11T18:44:51.476 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-objecter_ops": { 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 1024, 2024-03-11T18:44:51.477 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 48, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 48, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 48, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 48, 2024-03-11T18:44:51.478 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle": { 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:51.479 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 0, 2024-03-11T18:44:51.480 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 0, 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-write_buf_throttle-0x560df4672640": { 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:51.481 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 3758096384, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 9, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 2494, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 9, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:51.482 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 8, 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 2494, 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:51.483 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T18:44:52.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:51 smithi165 ceph-mon[36634]: pgmap v349: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 16 op/s 2024-03-11T18:44:52.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:51 smithi023 ceph-mon[36406]: pgmap v349: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 16 op/s 2024-03-11T18:44:52.218 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-11T18:44:52.218 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-11T18:44:52.218 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-11T18:44:52.218 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-11T18:44:52.218 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-11T18:44:52.218 DEBUG:tasks.check_counter:Getting stats from l 2024-03-11T18:44:52.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-11T18:44:52.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:51 smithi002 ceph-mon[31328]: pgmap v349: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 100 KiB/s wr, 16 op/s 2024-03-11T18:44:52.458 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T18:44:53.899 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:53 smithi165 ceph-mon[36634]: pgmap v350: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 91 KiB/s wr, 12 op/s 2024-03-11T18:44:54.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:53 smithi023 ceph-mon[36406]: pgmap v350: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 91 KiB/s wr, 12 op/s 2024-03-11T18:44:54.172 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-03-11T18:44:54.172 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-0": { 2024-03-11T18:44:54.172 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 4, 2024-03-11T18:44:54.172 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 4, 2024-03-11T18:44:54.172 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 3024, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 448, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 4, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 0, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.001301264, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.000036426, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.000504741, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:54.173 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 3, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000700242, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000233414 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.174 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 448 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-1": { 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 255, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 243, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 1709536, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 146336, 2024-03-11T18:44:54.175 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 2, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 1, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.050342367, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.013761291, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.041643280, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.005307285, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 242, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.010716220, 2024-03-11T18:44:54.176 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000044281 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 1709536, 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 146336 2024-03-11T18:44:54.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "AsyncMessenger::Worker-2": { 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_messages": 2, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages": 187, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_bytes": 320, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_bytes": 265376, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_created_connections": 6, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_active_connections": 3, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_total_time": 0.010843030, 2024-03-11T18:44:54.178 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_send_time": 0.007287736, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_recv_time": 0.001488014, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_messages_queue_lat": { 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 184, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.030589204, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000166245 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_handle_ack_lat": { 2024-03-11T18:44:54.179 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "msgr_send_encrypted_bytes": 265376 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "cct": { 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "total_workers": 0, 2024-03-11T18:44:54.180 INFO:teuthology.orchestra.run.smithi165.stdout: "unhealthy_workers": 0 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "mempool": { 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_bytes": 0, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bloom_filter_items": 0, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_bytes": 0, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_alloc_items": 0, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_bytes": 0, 2024-03-11T18:44:54.181 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_data_items": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_onode_items": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_meta_items": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_bytes": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_other_items": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-11T18:44:54.182 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_cache_buffer_items": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_bytes": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_extent_items": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_bytes": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_blob_items": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_shared_blob_items": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-11T18:44:54.183 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_inline_bl_items": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_bytes": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_fsck_items": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_bytes": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_txc_items": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_deferred_items": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_bytes": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluestore_writing_items": 0, 2024-03-11T18:44:54.184 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_bytes": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_items": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_bytes": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_reader_items": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_bytes": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "bluefs_file_writer_items": 0, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_bytes": 148732, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_anon_items": 31, 2024-03-11T18:44:54.185 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_bytes": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "buffer_meta_items": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_bytes": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_items": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_bytes": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_mapbl_items": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_bytes": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osd_pglog_items": 0, 2024-03-11T18:44:54.186 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_bytes": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_items": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_bytes": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "osdmap_mapping_items": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_bytes": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "pgmap_items": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_bytes": 928, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "mds_co_items": 2, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_bytes": 0, 2024-03-11T18:44:54.187 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_1_items": 0, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_bytes": 0, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "unittest_2_items": 0 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "val": 0, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "max": 104857600, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "get_started": 0, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "get": 257, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "get_sum": 1673591, 2024-03-11T18:44:54.188 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_fail": 0, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "get_or_fail_success": 257, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "take": 0, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "take_sum": 0, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "put": 257, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "put_sum": 1673591, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "wait": { 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "avgcount": 0, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "sum": 0.000000000, 2024-03-11T18:44:54.189 INFO:teuthology.orchestra.run.smithi165.stdout: "avgtime": 0.000000000 2024-03-11T18:44:54.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:54.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-03-11T18:44:54.190 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-03-11T18:44:54.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:53 smithi002 ceph-mon[31328]: pgmap v350: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 91 KiB/s wr, 12 op/s 2024-03-11T18:44:54.897 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-11T18:44:54.898 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-11T18:44:54.898 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-11T18:44:54.898 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-11T18:44:54.898 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-11T18:44:54.899 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-11T18:44:54.908 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-11T18:44:54.909 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:44:54.909 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T18:44:54.973 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-03-11T18:44:54.973 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T18:44:55.031 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:44:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.265+0000 7f104e400640 1 -- 172.21.15.2:0/11426549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 msgr2=0x7f1048105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.265+0000 7f104e400640 1 --2- 172.21.15.2:0/11426549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f1048105860 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1040009900 tx=0x7f104002bd10 comp rx=0 tx=0).stop 2024-03-11T18:44:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 -- 172.21.15.2:0/11426549 shutdown_connections 2024-03-11T18:44:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 --2- 172.21.15.2:0/11426549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f1048105860 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 --2- 172.21.15.2:0/11426549 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f1048104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 --2- 172.21.15.2:0/11426549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 -- 172.21.15.2:0/11426549 >> 172.21.15.2:0/11426549 conn(0x7f10480fb9e0 msgr2=0x7f10480fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 -- 172.21.15.2:0/11426549 shutdown_connections 2024-03-11T18:44:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.266+0000 7f104e400640 1 -- 172.21.15.2:0/11426549 wait complete. 2024-03-11T18:44:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.267+0000 7f104e400640 1 Processor -- start 2024-03-11T18:44:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.267+0000 7f104e400640 1 -- start start 2024-03-11T18:44:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f104819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104819e0d0 con 0x7f10481053d0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104819e240 con 0x7f10480fffd0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104e400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104819e400 con 0x7f10481008f0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f1047e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f104819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f104d000640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f1047e00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:50120/0 (socket says 172.21.15.2:50120) 2024-03-11T18:44:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.268+0000 7f1047e00640 1 -- 172.21.15.2:0/1417871414 learned_addr learned my addr 172.21.15.2:0/1417871414 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 -- 172.21.15.2:0/1417871414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 msgr2=0x7f10481a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 -- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 msgr2=0x7f104819da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f104819da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 -- 172.21.15.2:0/1417871414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1040009560 con 0x7f10481008f0 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f104d000640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:55.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1047e00640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f103000e9f0 tx=0x7f103000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030009670 con 0x7f10481008f0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1030005500 con 0x7f10481008f0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104819e6e0 con 0x7f10481008f0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030010430 con 0x7f10481008f0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.269+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10481aa970 con 0x7f10481008f0 2024-03-11T18:44:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.271+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104804ef50 con 0x7f10481008f0 2024-03-11T18:44:55.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.273+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f1030005020 con 0x7f10481008f0 2024-03-11T18:44:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.274+0000 7f1045600640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 0x7f102c0a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.274+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f1030014070 con 0x7f10481008f0 2024-03-11T18:44:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.274+0000 7f104d000640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 0x7f102c0a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.275+0000 7f104d000640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 0x7f102c0a40d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f104819ebc0 tx=0x7f1038009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:55.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.278+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1030096930 con 0x7f10481008f0 2024-03-11T18:44:55.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.480+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f104819efb0 con 0x7f10481008f0 2024-03-11T18:44:55.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.482+0000 7f1045600640 1 -- 172.21.15.2:0/1417871414 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+25349 (secure 0 0 0) 0x7f1030099ff0 con 0x7f10481008f0 2024-03-11T18:44:55.485 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:55.485 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":108,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:44:18.524871+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2218375343":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3512553041":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/492479102":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2651572933":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3571811795":"2024-03-12T18:33:42.314565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:44:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 msgr2=0x7f102c0a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 0x7f102c0a40d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f104819ebc0 tx=0x7f1038009500 comp rx=0 tx=0).stop 2024-03-11T18:44:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 msgr2=0x7f10481a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f103000e9f0 tx=0x7f103000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 shutdown_connections 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f102c0a1c10 0x7f102c0a40d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10481053d0 0x7f104819da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10481008f0 0x7f10481a3e10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 --2- 172.21.15.2:0/1417871414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10480fffd0 0x7f10481a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.485+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 >> 172.21.15.2:0/1417871414 conn(0x7f10480fb9e0 msgr2=0x7f1048102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.486+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 shutdown_connections 2024-03-11T18:44:55.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.486+0000 7f104e400640 1 -- 172.21.15.2:0/1417871414 wait complete. 2024-03-11T18:44:55.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T18:25:12.776324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-11T18:30:25.909135+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-11T18:30:27.217177+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-11T18:30:30.829620+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T18:44:55.510 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T18:44:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 -- 172.21.15.2:0/738767470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc10d950 msgr2=0x7f34cc10fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 --2- 172.21.15.2:0/738767470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc10d950 0x7f34cc10fda0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009ca0 tx=0x7f34c402c190 comp rx=0 tx=0).stop 2024-03-11T18:44:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 -- 172.21.15.2:0/738767470 shutdown_connections 2024-03-11T18:44:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 --2- 172.21.15.2:0/738767470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc10d950 0x7f34cc10fda0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 --2- 172.21.15.2:0/738767470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc10d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 --2- 172.21.15.2:0/738767470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc0735b0 0x7f34cc073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.739+0000 7f34d4200640 1 -- 172.21.15.2:0/738767470 >> 172.21.15.2:0/738767470 conn(0x7f34cc0fcc20 msgr2=0x7f34cc0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.740+0000 7f34d4200640 1 -- 172.21.15.2:0/738767470 shutdown_connections 2024-03-11T18:44:55.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.740+0000 7f34d4200640 1 -- 172.21.15.2:0/738767470 wait complete. 2024-03-11T18:44:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.740+0000 7f34d4200640 1 Processor -- start 2024-03-11T18:44:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 -- start start 2024-03-11T18:44:55.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0735b0 0x7f34cc19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc199ed0 con 0x7f34cc0735b0 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc19a040 con 0x7f34cc10d950 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d4200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc19a200 con 0x7f34cc073ed0 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d2200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.741+0000 7f34d2200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58790/0 (socket says 172.21.15.2:58790) 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d2200640 1 -- 172.21.15.2:0/1375476865 learned_addr learned my addr 172.21.15.2:0/1375476865 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:55.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d1800640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0735b0 0x7f34cc19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 -- 172.21.15.2:0/1375476865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 msgr2=0x7f34cc199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 -- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0735b0 msgr2=0x7f34cc19f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0735b0 0x7f34cc19f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 -- 172.21.15.2:0/1375476865 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4009900 con 0x7f34cc073ed0 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d2200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34d0e00640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc19fc00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34b800e9f0 tx=0x7f34b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:55.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.742+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b800ccc0 con 0x7f34cc073ed0 2024-03-11T18:44:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.743+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b800ce20 con 0x7f34cc073ed0 2024-03-11T18:44:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.743+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc19a4e0 con 0x7f34cc073ed0 2024-03-11T18:44:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.743+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8004a00 con 0x7f34cc073ed0 2024-03-11T18:44:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.743+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc1a66f0 con 0x7f34cc073ed0 2024-03-11T18:44:55.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.744+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc074c60 con 0x7f34cc073ed0 2024-03-11T18:44:55.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.747+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f34b80050b0 con 0x7f34cc073ed0 2024-03-11T18:44:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.748+0000 7f34ca000640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 0x7f34b40a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:55.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.748+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f34b8014070 con 0x7f34cc073ed0 2024-03-11T18:44:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.748+0000 7f34d1800640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 0x7f34b40a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.749+0000 7f34d1800640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 0x7f34b40a4060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009bd0 tx=0x7f34c0005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:55.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.749+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f34b8096720 con 0x7f34cc073ed0 2024-03-11T18:44:55.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.949+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34cc108ad0 con 0x7f34cc073ed0 2024-03-11T18:44:55.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.950+0000 7f34ca000640 1 -- 172.21.15.2:0/1375476865 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34b8099de0 con 0x7f34cc073ed0 2024-03-11T18:44:55.952 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-03-11T18:44:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.952+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 msgr2=0x7f34b40a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.952+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 0x7f34b40a4060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009bd0 tx=0x7f34c0005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.952+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 msgr2=0x7f34cc19fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.952+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc19fc00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f34b800e9f0 tx=0x7f34b800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 shutdown_connections 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34cc10d950 0x7f34cc199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34cc073ed0 0x7f34cc19fc00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f34b40a1ba0 0x7f34b40a4060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 --2- 172.21.15.2:0/1375476865 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0735b0 0x7f34cc19f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 >> 172.21.15.2:0/1375476865 conn(0x7f34cc0fcc20 msgr2=0x7f34cc0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 shutdown_connections 2024-03-11T18:44:55.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:55.953+0000 7f34d4200640 1 -- 172.21.15.2:0/1375476865 wait complete. 2024-03-11T18:44:55.974 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-11T18:44:56.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:55 smithi165 ceph-mon[36634]: pgmap v351: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 15 op/s 2024-03-11T18:44:56.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:55 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1417871414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:56.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:55 smithi023 ceph-mon[36406]: pgmap v351: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 15 op/s 2024-03-11T18:44:56.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:55 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1417871414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:56.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.198+0000 7fccdc600640 1 -- 172.21.15.2:0/3781338835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 msgr2=0x7fccd4103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.198+0000 7fccdc600640 1 --2- 172.21.15.2:0/3781338835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd4103990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009890 tx=0x7fccc802bb70 comp rx=0 tx=0).stop 2024-03-11T18:44:56.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 -- 172.21.15.2:0/3781338835 shutdown_connections 2024-03-11T18:44:56.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 --2- 172.21.15.2:0/3781338835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4108680 0x7fccd4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 --2- 172.21.15.2:0/3781338835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4103f60 0x7fccd4107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 --2- 172.21.15.2:0/3781338835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd4103990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 -- 172.21.15.2:0/3781338835 >> 172.21.15.2:0/3781338835 conn(0x7fccd40fedf0 msgr2=0x7fccd4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:56.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 -- 172.21.15.2:0/3781338835 shutdown_connections 2024-03-11T18:44:56.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.199+0000 7fccdc600640 1 -- 172.21.15.2:0/3781338835 wait complete. 2024-03-11T18:44:56.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 Processor -- start 2024-03-11T18:44:56.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 -- start start 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4108680 0x7fccd419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4199900 0x7fccd4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd419a2f0 con 0x7fccd4199900 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd419a460 con 0x7fccd41035b0 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.200+0000 7fccdc600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd419a620 con 0x7fccd4108680 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccda600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4199900 0x7fccd4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58810/0 (socket says 172.21.15.2:58810) 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 -- 172.21.15.2:0/1827061519 learned_addr learned my addr 172.21.15.2:0/1827061519 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9200640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4108680 0x7fccd419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 -- 172.21.15.2:0/1827061519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4108680 msgr2=0x7fccd419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4108680 0x7fccd419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 -- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4199900 msgr2=0x7fccd4199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4199900 0x7fccd4199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.201+0000 7fccd9c00640 1 -- 172.21.15.2:0/1827061519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc009900 con 0x7fccd41035b0 2024-03-11T18:44:56.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccd9c00640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fccc8000c00 tx=0x7fccc8004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:56.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8034950 con 0x7fccd41035b0 2024-03-11T18:44:56.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccc802c9f0 con 0x7fccd41035b0 2024-03-11T18:44:56.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8032600 con 0x7fccd41035b0 2024-03-11T18:44:56.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc8009490 con 0x7fccd41035b0 2024-03-11T18:44:56.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.202+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd41a67f0 con 0x7fccd41035b0 2024-03-11T18:44:56.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.203+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd4068450 con 0x7fccd41035b0 2024-03-11T18:44:56.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.205+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fccc803b040 con 0x7fccd41035b0 2024-03-11T18:44:56.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.206+0000 7fccd2000640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 0x7fccbc0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.206+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fccc80ef5d0 con 0x7fccd41035b0 2024-03-11T18:44:56.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.207+0000 7fccd9200640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 0x7fccbc0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.207+0000 7fccd9200640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 0x7fccbc0a40e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fccd4102660 tx=0x7fccc0005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:56.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.211+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fccc80b81e0 con 0x7fccd41035b0 2024-03-11T18:44:56.252 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:55 smithi002 ceph-mon[31328]: pgmap v351: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 91 KiB/s wr, 15 op/s 2024-03-11T18:44:56.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:55 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1417871414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:44:56.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.416+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fccd410b8b0 con 0x7fccd41035b0 2024-03-11T18:44:56.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.417+0000 7fccd2000640 1 -- 172.21.15.2:0/1827061519 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7fccc80bb8a0 con 0x7fccd41035b0 2024-03-11T18:44:56.419 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:44:56.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.419+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 msgr2=0x7fccbc0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.419+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 0x7fccbc0a40e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fccd4102660 tx=0x7fccc0005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:56.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.419+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 msgr2=0x7fccd419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.419+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fccc8000c00 tx=0x7fccc8004530 comp rx=0 tx=0).stop 2024-03-11T18:44:56.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 shutdown_connections 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fccbc0a1c20 0x7fccbc0a40e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccd4199900 0x7fccd4199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd4108680 0x7fccd419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 --2- 172.21.15.2:0/1827061519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccd41035b0 0x7fccd419f750 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 >> 172.21.15.2:0/1827061519 conn(0x7fccd40fedf0 msgr2=0x7fccd4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 shutdown_connections 2024-03-11T18:44:56.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.420+0000 7fccdc600640 1 -- 172.21.15.2:0/1827061519 wait complete. 2024-03-11T18:44:56.443 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 -- 172.21.15.2:0/4179679923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 msgr2=0x7f85d0105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 --2- 172.21.15.2:0/4179679923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d0105770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009900 tx=0x7f85c802bcc0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 -- 172.21.15.2:0/4179679923 shutdown_connections 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 --2- 172.21.15.2:0/4179679923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d0105770 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 --2- 172.21.15.2:0/4179679923 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 0x7f85d0104ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 --2- 172.21.15.2:0/4179679923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.670+0000 7f85d4c00640 1 -- 172.21.15.2:0/4179679923 >> 172.21.15.2:0/4179679923 conn(0x7f85d0076c10 msgr2=0x7f85d00fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.671+0000 7f85d4c00640 1 -- 172.21.15.2:0/4179679923 shutdown_connections 2024-03-11T18:44:56.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.671+0000 7f85d4c00640 1 -- 172.21.15.2:0/4179679923 wait complete. 2024-03-11T18:44:56.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.671+0000 7f85d4c00640 1 Processor -- start 2024-03-11T18:44:56.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 -- start start 2024-03-11T18:44:56.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 0x7f85d01a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d019dfc0 con 0x7f85d0105300 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d019e130 con 0x7f85d00fff00 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85d4c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d019e2f0 con 0x7f85d0100820 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85cf400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85cf400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:58826/0 (socket says 172.21.15.2:58826) 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85cf400640 1 -- 172.21.15.2:0/1291259192 learned_addr learned my addr 172.21.15.2:0/1291259192 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.672+0000 7f85cea00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 0x7f85d01a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cfe00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 -- 172.21.15.2:0/1291259192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 msgr2=0x7f85d01a3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 0x7f85d01a3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 -- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 msgr2=0x7f85d019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 -- 172.21.15.2:0/1291259192 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009560 con 0x7f85d00fff00 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cfe00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85cf400640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f85c000e9f0 tx=0x7f85c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0009700 con 0x7f85d00fff00 2024-03-11T18:44:56.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d019e5d0 con 0x7f85d00fff00 2024-03-11T18:44:56.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.673+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85c000cd80 con 0x7f85d00fff00 2024-03-11T18:44:56.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.674+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0018920 con 0x7f85d00fff00 2024-03-11T18:44:56.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.674+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d01aa860 con 0x7f85d00fff00 2024-03-11T18:44:56.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.675+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d004eee0 con 0x7f85d00fff00 2024-03-11T18:44:56.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.677+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f85c0018a80 con 0x7f85d00fff00 2024-03-11T18:44:56.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.678+0000 7f85c7e00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 0x7f85b00a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:56.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.678+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f85c0014070 con 0x7f85d00fff00 2024-03-11T18:44:56.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.678+0000 7f85cea00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 0x7f85b00a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:56.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.679+0000 7f85cea00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 0x7f85b00a40e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85d0105a40 tx=0x7f85b8009500 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:56.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.681+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f85c0096ef0 con 0x7f85d00fff00 2024-03-11T18:44:56.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.885+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85d01176f0 con 0x7f85d00fff00 2024-03-11T18:44:56.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.886+0000 7f85c7e00640 1 -- 172.21.15.2:0/1291259192 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f85c00260a0 con 0x7f85d00fff00 2024-03-11T18:44:56.889 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-03-11T18:44:56.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 msgr2=0x7f85b00a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 0x7f85b00a40e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85d0105a40 tx=0x7f85b8009500 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 msgr2=0x7f85d01a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f85c000e9f0 tx=0x7f85c000eeb0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 shutdown_connections 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f85b00a1c20 0x7f85b00a40e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85d0105300 0x7f85d019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85d0100820 0x7f85d01a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 --2- 172.21.15.2:0/1291259192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85d00fff00 0x7f85d01a37c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 >> 172.21.15.2:0/1291259192 conn(0x7f85d0076c10 msgr2=0x7f85d0102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:56.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 shutdown_connections 2024-03-11T18:44:56.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:56.889+0000 7f85d4c00640 1 -- 172.21.15.2:0/1291259192 wait complete. 2024-03-11T18:44:56.911 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-11T18:44:57.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 -- 172.21.15.2:0/3706530163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc101190 msgr2=0x7f7bbc101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 --2- 172.21.15.2:0/3706530163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc101190 0x7f7bbc101570 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009900 tx=0x7f7bac02bc30 comp rx=0 tx=0).stop 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 -- 172.21.15.2:0/3706530163 shutdown_connections 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 --2- 172.21.15.2:0/3706530163 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc106260 0x7f7bbc10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 --2- 172.21.15.2:0/3706530163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101b40 0x7f7bbc105b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 --2- 172.21.15.2:0/3706530163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc101190 0x7f7bbc101570 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 -- 172.21.15.2:0/3706530163 >> 172.21.15.2:0/3706530163 conn(0x7f7bbc0fca30 msgr2=0x7f7bbc0fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.134+0000 7f7bc1600640 1 -- 172.21.15.2:0/3706530163 shutdown_connections 2024-03-11T18:44:57.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.135+0000 7f7bc1600640 1 -- 172.21.15.2:0/3706530163 wait complete. 2024-03-11T18:44:57.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 Processor -- start 2024-03-11T18:44:57.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 -- start start 2024-03-11T18:44:57.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc106260 0x7f7bbc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc19bea0 0x7f7bbc19c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19c890 con 0x7f7bbc19bea0 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19ca00 con 0x7f7bbc101190 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.136+0000 7f7bc1600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19cb70 con 0x7f7bbc106260 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:36138/0 (socket says 172.21.15.2:36138) 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 -- 172.21.15.2:0/1514285629 learned_addr learned my addr 172.21.15.2:0/1514285629 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:57.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bc0c00640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc19bea0 0x7f7bbc19c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbb400640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc106260 0x7f7bbc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 -- 172.21.15.2:0/1514285629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc106260 msgr2=0x7f7bbc197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc106260 0x7f7bbc197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 -- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc19bea0 msgr2=0x7f7bbc19c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc19bea0 0x7f7bbc19c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.137+0000 7f7bbbe00640 1 -- 172.21.15.2:0/1514285629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac009560 con 0x7f7bbc101190 2024-03-11T18:44:57.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bbbe00640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bac005940 tx=0x7f7bac02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:57.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0042d0 con 0x7f7bbc101190 2024-03-11T18:44:57.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bac004430 con 0x7f7bbc101190 2024-03-11T18:44:57.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc1a4150 con 0x7f7bbc101190 2024-03-11T18:44:57.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac003c50 con 0x7f7bbc101190 2024-03-11T18:44:57.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.138+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc1a45d0 con 0x7f7bbc101190 2024-03-11T18:44:57.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.139+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc04eee0 con 0x7f7bbc101190 2024-03-11T18:44:57.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.141+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f7bac0054f0 con 0x7f7bbc101190 2024-03-11T18:44:57.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.142+0000 7f7bb8c00640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 0x7f7ba00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.143+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f7bac0efe70 con 0x7f7bbc101190 2024-03-11T18:44:57.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.143+0000 7f7bbb400640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 0x7f7ba00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.144+0000 7f7bbb400640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 0x7f7ba00a4060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc100160 tx=0x7f7ba8005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:57.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.145+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7bac0b89d0 con 0x7f7bbc101190 2024-03-11T18:44:57.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:56 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1375476865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:56 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1827061519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:56 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1375476865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:56 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1827061519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:56 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1375476865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:56 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1827061519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-11T18:44:57.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.346+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7bbc1094f0 con 0x7f7bbc101190 2024-03-11T18:44:57.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.346+0000 7f7bb8c00640 1 -- 172.21.15.2:0/1514285629 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v108) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7bac0bc090 con 0x7f7bbc101190 2024-03-11T18:44:57.349 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-03-11T18:44:57.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.349+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 msgr2=0x7f7ba00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 0x7f7ba00a4060 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc100160 tx=0x7f7ba8005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:57.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 msgr2=0x7f7bbc1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7bac005940 tx=0x7f7bac02c670 comp rx=0 tx=0).stop 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 shutdown_connections 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f7ba00a1ba0 0x7f7ba00a4060 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bbc19bea0 0x7f7bbc19c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7bbc106260 0x7f7bbc197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 --2- 172.21.15.2:0/1514285629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc101190 0x7f7bbc1975e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 >> 172.21.15.2:0/1514285629 conn(0x7f7bbc0fca30 msgr2=0x7f7bbc0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 shutdown_connections 2024-03-11T18:44:57.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.350+0000 7f7bc1600640 1 -- 172.21.15.2:0/1514285629 wait complete. 2024-03-11T18:44:57.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-11T18:44:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 -- 172.21.15.2:0/818778190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 msgr2=0x7fe0dc114430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 --2- 172.21.15.2:0/818778190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc114430 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4009900 tx=0x7fe0d402bcd0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 -- 172.21.15.2:0/818778190 shutdown_connections 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 --2- 172.21.15.2:0/818778190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc114430 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 --2- 172.21.15.2:0/818778190 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc107580 0x7fe0dc107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 --2- 172.21.15.2:0/818778190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc10c540 0x7fe0dc10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 -- 172.21.15.2:0/818778190 >> 172.21.15.2:0/818778190 conn(0x7fe0dc076d80 msgr2=0x7fe0dc0791a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 -- 172.21.15.2:0/818778190 shutdown_connections 2024-03-11T18:44:57.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.590+0000 7fe0e1600640 1 -- 172.21.15.2:0/818778190 wait complete. 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.591+0000 7fe0e1600640 1 Processor -- start 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.591+0000 7fe0e1600640 1 -- start start 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.591+0000 7fe0e1600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc107580 0x7fe0dc1a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.591+0000 7fe0e1600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc1a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e1600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc19e020 con 0x7fe0dc10c540 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e1600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc19e190 con 0x7fe0dc107f40 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e1600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc19e350 con 0x7fe0dc107580 2024-03-11T18:44:57.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55192/0 (socket says 172.21.15.2:55192) 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 -- 172.21.15.2:0/1705961377 learned_addr learned my addr 172.21.15.2:0/1705961377 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0db400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc1a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0dbe00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc107580 0x7fe0dc1a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 -- 172.21.15.2:0/1705961377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc107580 msgr2=0x7fe0dc1a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc107580 0x7fe0dc1a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 -- 172.21.15.2:0/1705961377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 msgr2=0x7fe0dc1a3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc1a3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0e0c00640 1 -- 172.21.15.2:0/1705961377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4009840 con 0x7fe0dc10c540 2024-03-11T18:44:57.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.592+0000 7fe0db400640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc1a3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0e0c00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe0d402c210 tx=0x7fe0d40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d402dbe0 con 0x7fe0dc10c540 2024-03-11T18:44:57.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d4009560 con 0x7fe0dc10c540 2024-03-11T18:44:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0d402dd40 con 0x7fe0dc10c540 2024-03-11T18:44:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4003b50 con 0x7fe0dc10c540 2024-03-11T18:44:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.593+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc1aa6e0 con 0x7fe0dc10c540 2024-03-11T18:44:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.595+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc04eee0 con 0x7fe0dc10c540 2024-03-11T18:44:57.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.597+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fe0d4003cb0 con 0x7fe0dc10c540 2024-03-11T18:44:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.598+0000 7fe0d8c00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 0x7fe0c00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.598+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7fe0d40f01b0 con 0x7fe0dc10c540 2024-03-11T18:44:57.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.598+0000 7fe0dbe00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 0x7fe0c00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:57.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.599+0000 7fe0dbe00640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 0x7fe0c00a4060 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc19eb10 tx=0x7fe0cc005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:57.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.603+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe0d40b8d10 con 0x7fe0dc10c540 2024-03-11T18:44:57.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.817+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe0dc19ef00 con 0x7fe0dc10c540 2024-03-11T18:44:57.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.818+0000 7fe0d8c00640 1 -- 172.21.15.2:0/1705961377 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v108) v1 ==== 81+0+985 (secure 0 0 0) 0x7fe0d40433f0 con 0x7fe0dc10c540 2024-03-11T18:44:57.820 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3303291273 2024-03-12T18:22:28.050188+0000 2024-03-11T18:44:57.820 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1446641624 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.820 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2218375343 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3512553041 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3920276264 2024-03-12T18:22:28.050188+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/247107605 2024-03-12T18:21:53.065562+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3822870648 2024-03-12T18:21:53.065562+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/1969024355 2024-03-12T18:22:28.050188+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:44:57.821 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/1779941333 2024-03-12T18:21:53.065562+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6801/2981412802 2024-03-12T18:22:28.050188+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3359009707 2024-03-12T18:21:53.065562+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/492479102 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/2651572933 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6800/1446641624 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.822 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:0/3571811795 2024-03-12T18:33:42.314565+0000 2024-03-11T18:44:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:listed 17 entries 2024-03-11T18:44:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 msgr2=0x7fe0c00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 0x7fe0c00a4060 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc19eb10 tx=0x7fe0cc005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 msgr2=0x7fe0dc19d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe0d402c210 tx=0x7fe0d40045e0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 shutdown_connections 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fe0c00a1ba0 0x7fe0c00a4060 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0dc10c540 0x7fe0dc19d9d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0dc107f40 0x7fe0dc1a3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 --2- 172.21.15.2:0/1705961377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe0dc107580 0x7fe0dc1a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.820+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 >> 172.21.15.2:0/1705961377 conn(0x7fe0dc076d80 msgr2=0x7fe0dc077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.821+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 shutdown_connections 2024-03-11T18:44:57.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:57.821+0000 7fe0e1600640 1 -- 172.21.15.2:0/1705961377 wait complete. 2024-03-11T18:44:57.844 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-11T18:44:57.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:44:57.844 DEBUG:teuthology.orchestra.run.smithi002:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-11T18:44:58.010 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:44:58.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:44:58.010 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-11T18:44:58.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:44:58.093 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-03-11T18:44:58.138 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-11T18:44:58.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:44:58.139 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T18:44:58.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:57 smithi165 ceph-mon[36634]: pgmap v352: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-03-11T18:44:58.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:57 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1291259192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:57 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1514285629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:57 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1705961377' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:44:58.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:57 smithi023 ceph-mon[36406]: pgmap v352: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-03-11T18:44:58.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:57 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1291259192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:57 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1514285629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:57 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1705961377' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:44:58.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:57 smithi002 ceph-mon[31328]: pgmap v352: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2024-03-11T18:44:58.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:57 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1291259192' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:57 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1514285629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-11T18:44:58.242 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:57 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1705961377' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T18:44:58.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T18:44:58.253 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2024-03-11T18:44:58.361 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T18:44:58.370 INFO:tasks.cephadm:Teardown begin 2024-03-11T18:44:58.371 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-11T18:44:58.371 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:44:58.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 -- 172.21.15.2:0/1815989604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4103e60 msgr2=0x7f8fe4107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:58.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/1815989604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4103e60 0x7f8fe4107ea0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009940 tx=0x7f8fcc02bc10 comp rx=0 tx=0).stop 2024-03-11T18:44:58.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 -- 172.21.15.2:0/1815989604 shutdown_connections 2024-03-11T18:44:58.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/1815989604 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe4108580 0x7f8fe4110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/1815989604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4103e60 0x7f8fe4107ea0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/1815989604 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe41034b0 0x7f8fe4103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 -- 172.21.15.2:0/1815989604 >> 172.21.15.2:0/1815989604 conn(0x7f8fe40fecb0 msgr2=0x7f8fe41010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 -- 172.21.15.2:0/1815989604 shutdown_connections 2024-03-11T18:44:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.567+0000 7f8fe8c00640 1 -- 172.21.15.2:0/1815989604 wait complete. 2024-03-11T18:44:58.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.568+0000 7f8fe8c00640 1 Processor -- start 2024-03-11T18:44:58.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 -- start start 2024-03-11T18:44:58.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:58.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 0x7f8fe419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:58.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 0x7f8fe4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe419a1f0 con 0x7f8fe41034b0 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe419a360 con 0x7f8fe4199720 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe8c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe419a520 con 0x7f8fe4108580 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55206/0 (socket says 172.21.15.2:55206) 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fe3400640 1 -- 172.21.15.2:0/2217927347 learned_addr learned my addr 172.21.15.2:0/2217927347 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.569+0000 7f8fdaa00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 0x7f8fe419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fe3e00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 0x7f8fe4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 -- 172.21.15.2:0/2217927347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 msgr2=0x7f8fe4199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:58.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 0x7f8fe4199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 -- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 msgr2=0x7f8fe419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 -- 172.21.15.2:0/2217927347 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc009500 con 0x7f8fe4108580 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fe3e00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 0x7f8fe4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fe3400640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:44:58.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.570+0000 7f8fdaa00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 0x7f8fe419fba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0097b0 tx=0x7f8fcc034ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.571+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc034990 con 0x7f8fe4108580 2024-03-11T18:44:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.571+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe41a6460 con 0x7f8fe4108580 2024-03-11T18:44:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.571+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc005530 con 0x7f8fe4108580 2024-03-11T18:44:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.571+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc02cc90 con 0x7f8fe4108580 2024-03-11T18:44:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.571+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe41a6930 con 0x7f8fe4108580 2024-03-11T18:44:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.572+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4104980 con 0x7f8fe4108580 2024-03-11T18:44:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.574+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f8fcc02c5b0 con 0x7f8fe4108580 2024-03-11T18:44:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.575+0000 7f8fe0c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 0x7f8fc80a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.575+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f8fcc0eff10 con 0x7f8fe4108580 2024-03-11T18:44:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.576+0000 7f8fe3400640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 0x7f8fc80a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.576+0000 7f8fe3400640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 0x7f8fc80a40d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4011550 tx=0x7f8fd4005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:58.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.578+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8fcc0b8a70 con 0x7f8fe4108580 2024-03-11T18:44:58.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.821+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fe410b7b0 con 0x7f8fe4108580 2024-03-11T18:44:58.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.822+0000 7f8fe0c00640 1 -- 172.21.15.2:0/2217927347 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11332 (secure 0 0 0) 0x7f8fcc0bc130 con 0x7f8fe4108580 2024-03-11T18:44:58.825 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:58.825 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:41:36.671984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/1442654134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1442654134},{"type":"v1","addr":"172.21.15.165:6837","nonce":1442654134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:44:58.825 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 31 2024-03-11T18:44:58.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 msgr2=0x7f8fc80a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 0x7f8fc80a40d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4011550 tx=0x7f8fd4005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 msgr2=0x7f8fe419fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 0x7f8fe419fba0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0097b0 tx=0x7f8fcc034ee0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 shutdown_connections 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8fe4199720 0x7f8fe4199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8fe4108580 0x7f8fe419fba0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8fc80a1c10 0x7f8fc80a40d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 --2- 172.21.15.2:0/2217927347 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fe41034b0 0x7f8fe419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:58.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 >> 172.21.15.2:0/2217927347 conn(0x7f8fe40fecb0 msgr2=0x7f8fe4100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:58.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 shutdown_connections 2024-03-11T18:44:58.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:58.825+0000 7f8fe8c00640 1 -- 172.21.15.2:0/2217927347 wait complete. 2024-03-11T18:44:58.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T18:44:59.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 -- 172.21.15.2:0/182656371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff964108580 msgr2=0x7ff964110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 --2- 172.21.15.2:0/182656371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff964108580 0x7ff964110150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009cd0 tx=0x7ff95c02c0f0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 -- 172.21.15.2:0/182656371 shutdown_connections 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 --2- 172.21.15.2:0/182656371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff964108580 0x7ff964110150 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 --2- 172.21.15.2:0/182656371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964103e60 0x7ff964107ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 --2- 172.21.15.2:0/182656371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff9641034b0 0x7ff964103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.067+0000 7ff968c00640 1 -- 172.21.15.2:0/182656371 >> 172.21.15.2:0/182656371 conn(0x7ff9640fecb0 msgr2=0x7ff9641010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.068+0000 7ff968c00640 1 -- 172.21.15.2:0/182656371 shutdown_connections 2024-03-11T18:44:59.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.068+0000 7ff968c00640 1 -- 172.21.15.2:0/182656371 wait complete. 2024-03-11T18:44:59.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 Processor -- start 2024-03-11T18:44:59.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 -- start start 2024-03-11T18:44:59.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff964108580 0x7ff96419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964199690 0x7ff964199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96419a160 con 0x7ff964199690 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96419a2d0 con 0x7ff9641034b0 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.069+0000 7ff968c00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96419a490 con 0x7ff964108580 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff95aa00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff964108580 0x7ff96419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964199690 0x7ff964199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:36184/0 (socket says 172.21.15.2:36184) 2024-03-11T18:44:59.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 -- 172.21.15.2:0/3230625665 learned_addr learned my addr 172.21.15.2:0/3230625665 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 -- 172.21.15.2:0/3230625665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff964108580 msgr2=0x7ff96419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff964108580 0x7ff96419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 -- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964199690 msgr2=0x7ff964199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964199690 0x7ff964199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 -- 172.21.15.2:0/3230625665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c009900 con 0x7ff9641034b0 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.070+0000 7ff963400640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff95400e9f0 tx=0x7ff95400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.071+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954009720 con 0x7ff9641034b0 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.071+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff95400cd80 con 0x7ff9641034b0 2024-03-11T18:44:59.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.071+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9641a6430 con 0x7ff9641034b0 2024-03-11T18:44:59.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.071+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954010690 con 0x7ff9641034b0 2024-03-11T18:44:59.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.071+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9641a6910 con 0x7ff9641034b0 2024-03-11T18:44:59.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.072+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff964104980 con 0x7ff9641034b0 2024-03-11T18:44:59.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.074+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7ff9540046c0 con 0x7ff9641034b0 2024-03-11T18:44:59.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.075+0000 7ff960c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 0x7ff9480a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.075+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7ff954014070 con 0x7ff9641034b0 2024-03-11T18:44:59.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.075+0000 7ff95aa00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 0x7ff9480a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.076+0000 7ff95aa00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 0x7ff9480a40d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff96419aee0 tx=0x7ff94c005f30 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:59.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.079+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff9540d2070 con 0x7ff9641034b0 2024-03-11T18:44:59.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:58 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/2217927347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:44:59.156 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:58 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/2217927347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:44:59.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:58 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/2217927347' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:44:59.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.325+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff96410b810 con 0x7ff9641034b0 2024-03-11T18:44:59.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.326+0000 7ff960c00640 1 -- 172.21.15.2:0/3230625665 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11332 (secure 0 0 0) 0x7ff9540969c0 con 0x7ff9641034b0 2024-03-11T18:44:59.329 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:59.329 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2188580308","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2188580308},{"type":"v1","addr":"172.21.15.2:6835","nonce":2188580308}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6839/3374917731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6838","nonce":3374917731},{"type":"v1","addr":"172.21.15.2:6839","nonce":3374917731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14478,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6841/2401819833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6840","nonce":2401819833},{"type":"v1","addr":"172.21.15.2:6841","nonce":2401819833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.165:6833/4285430616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6832","nonce":4285430616},{"type":"v1","addr":"172.21.15.165:6833","nonce":4285430616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6833/2120271757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2120271757},{"type":"v1","addr":"172.21.15.23:6833","nonce":2120271757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6837/2729524869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6836","nonce":2729524869},{"type":"v1","addr":"172.21.15.2:6837","nonce":2729524869}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T18:30:27.790111+0000","modified":"2024-03-11T18:41:36.671984+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24550,"mds_2":24521},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6835/1605486605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1605486605},{"type":"v1","addr":"172.21.15.23:6835","nonce":1605486605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6835/2758205355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6834","nonce":2758205355},{"type":"v1","addr":"172.21.15.165:6835","nonce":2758205355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24521":{"gid":24521,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.23:6837/531049865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":531049865},{"type":"v1","addr":"172.21.15.23:6837","nonce":531049865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.165:6839/1329073865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6838","nonce":1329073865},{"type":"v1","addr":"172.21.15.165:6839","nonce":1329073865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6839/2250151518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":2250151518},{"type":"v1","addr":"172.21.15.23:6839","nonce":2250151518}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_26113":{"gid":26113,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.165:6837/1442654134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6836","nonce":1442654134},{"type":"v1","addr":"172.21.15.165:6837","nonce":1442654134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-11T18:44:59.329 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 31 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 msgr2=0x7ff9480a40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 0x7ff9480a40d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff96419aee0 tx=0x7ff94c005f30 comp rx=0 tx=0).stop 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 msgr2=0x7ff96419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff95400e9f0 tx=0x7ff95400edf0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 shutdown_connections 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7ff9480a1c10 0x7ff9480a40d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff964199690 0x7ff964199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff964108580 0x7ff96419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 --2- 172.21.15.2:0/3230625665 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9641034b0 0x7ff96419f5d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.329+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 >> 172.21.15.2:0/3230625665 conn(0x7ff9640fecb0 msgr2=0x7ff9640ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:59.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.330+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 shutdown_connections 2024-03-11T18:44:59.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.330+0000 7ff968c00640 1 -- 172.21.15.2:0/3230625665 wait complete. 2024-03-11T18:44:59.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T18:44:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 -- 172.21.15.2:0/2393549010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148106450 msgr2=0x7f714810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 --2- 172.21.15.2:0/2393549010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148106450 0x7f714810e020 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7140009cd0 tx=0x7f714002c0f0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 -- 172.21.15.2:0/2393549010 shutdown_connections 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 --2- 172.21.15.2:0/2393549010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148106450 0x7f714810e020 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 --2- 172.21.15.2:0/2393549010 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101d30 0x7f7148105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 --2- 172.21.15.2:0/2393549010 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148101380 0x7f7148101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 -- 172.21.15.2:0/2393549010 >> 172.21.15.2:0/2393549010 conn(0x7f71480fcc20 msgr2=0x7f71480ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 -- 172.21.15.2:0/2393549010 shutdown_connections 2024-03-11T18:44:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.573+0000 7f714f400640 1 -- 172.21.15.2:0/2393549010 wait complete. 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 Processor -- start 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 -- start start 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148106450 0x7f714819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148197630 0x7f7148197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148198100 con 0x7f7148106450 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148198270 con 0x7f7148197630 2024-03-11T18:44:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f714f400640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148198430 con 0x7f7148101380 2024-03-11T18:44:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.575+0000 7f7147e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148106450 0x7f714819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.2:38990/0 (socket says 172.21.15.2:38990) 2024-03-11T18:44:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 -- 172.21.15.2:0/718547149 learned_addr learned my addr 172.21.15.2:0/718547149 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:44:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714d400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148197630 0x7f7148197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 -- 172.21.15.2:0/718547149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148197630 msgr2=0x7f7148197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148197630 0x7f7148197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 -- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148106450 msgr2=0x7f714819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148106450 0x7f714819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 -- 172.21.15.2:0/718547149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140009900 con 0x7f7148101380 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.576+0000 7f714ca00640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f713800e9f0 tx=0x7f713800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.577+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138009700 con 0x7f7148101380 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.577+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71481a43d0 con 0x7f7148101380 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.577+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f713800cd80 con 0x7f7148101380 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.577+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138018920 con 0x7f7148101380 2024-03-11T18:44:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.577+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71481a4900 con 0x7f7148101380 2024-03-11T18:44:59.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.578+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148102850 con 0x7f7148101380 2024-03-11T18:44:59.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.580+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f7138018a80 con 0x7f7148101380 2024-03-11T18:44:59.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.581+0000 7f7145600640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 0x7f712c0a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:44:59.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.582+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f7138014070 con 0x7f7148101380 2024-03-11T18:44:59.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.582+0000 7f7147e00640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 0x7f712c0a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:44:59.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.582+0000 7f7147e00640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 0x7f712c0a40e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7148198e80 tx=0x7f7130005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:44:59.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.585+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7138052d20 con 0x7f7148101380 2024-03-11T18:44:59.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.787+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71481096e0 con 0x7f7148101380 2024-03-11T18:44:59.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.790+0000 7f7145600640 1 -- 172.21.15.2:0/718547149 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+25349 (secure 0 0 0) 0x7f7138096a60 con 0x7f7148101380 2024-03-11T18:44:59.793 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-03-11T18:44:59.793 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":108,"fsid":"02b83714-dfd4-11ee-95c8-87774f69a715","created":"2024-03-11T18:21:20.287842+0000","modified":"2024-03-11T18:44:18.524871+0000","last_up_change":"2024-03-11T18:29:29.808784+0000","last_in_change":"2024-03-11T18:29:10.028834+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T18:25:12.776324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-11T18:30:25.909135+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-11T18:30:27.217177+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-11T18:30:30.829620+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a773fb9c-07bb-4b3e-94d4-8c382a21561e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6803","nonce":1098127519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6805","nonce":1098127519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6809","nonce":1098127519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":1098127519},{"type":"v1","addr":"172.21.15.2:6807","nonce":1098127519}]},"public_addr":"172.21.15.2:6803/1098127519","cluster_addr":"172.21.15.2:6805/1098127519","heartbeat_back_addr":"172.21.15.2:6809/1098127519","heartbeat_front_addr":"172.21.15.2:6807/1098127519","state":["exists","up"]},{"osd":1,"uuid":"ec6ef905-1b64-44d7-bbd0-d52bc91b22df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6801","nonce":1383496716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6803","nonce":1383496716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6807","nonce":1383496716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":1383496716},{"type":"v1","addr":"172.21.15.23:6805","nonce":1383496716}]},"public_addr":"172.21.15.23:6801/1383496716","cluster_addr":"172.21.15.23:6803/1383496716","heartbeat_back_addr":"172.21.15.23:6807/1383496716","heartbeat_front_addr":"172.21.15.23:6805/1383496716","state":["exists","up"]},{"osd":2,"uuid":"fa7a2e4d-c63a-479a-b306-c3c983dea728","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6801","nonce":758382799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6803","nonce":758382799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6807","nonce":758382799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":758382799},{"type":"v1","addr":"172.21.15.165:6805","nonce":758382799}]},"public_addr":"172.21.15.165:6801/758382799","cluster_addr":"172.21.15.165:6803/758382799","heartbeat_back_addr":"172.21.15.165:6807/758382799","heartbeat_front_addr":"172.21.15.165:6805/758382799","state":["exists","up"]},{"osd":3,"uuid":"332dc7bc-07d1-4637-b31b-8c2e2171d1da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6811","nonce":2581962705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6813","nonce":2581962705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6817","nonce":2581962705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2581962705},{"type":"v1","addr":"172.21.15.2:6815","nonce":2581962705}]},"public_addr":"172.21.15.2:6811/2581962705","cluster_addr":"172.21.15.2:6813/2581962705","heartbeat_back_addr":"172.21.15.2:6817/2581962705","heartbeat_front_addr":"172.21.15.2:6815/2581962705","state":["exists","up"]},{"osd":4,"uuid":"1fa522c2-c663-4c43-b022-ce54054575ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6809","nonce":2973022110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6811","nonce":2973022110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6815","nonce":2973022110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":2973022110},{"type":"v1","addr":"172.21.15.23:6813","nonce":2973022110}]},"public_addr":"172.21.15.23:6809/2973022110","cluster_addr":"172.21.15.23:6811/2973022110","heartbeat_back_addr":"172.21.15.23:6815/2973022110","heartbeat_front_addr":"172.21.15.23:6813/2973022110","state":["exists","up"]},{"osd":5,"uuid":"c0f4c30d-2ccb-4ed3-8562-a935e88af3b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6809","nonce":1264754008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6811","nonce":1264754008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6815","nonce":1264754008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":1264754008},{"type":"v1","addr":"172.21.15.165:6813","nonce":1264754008}]},"public_addr":"172.21.15.165:6809/1264754008","cluster_addr":"172.21.15.165:6811/1264754008","heartbeat_back_addr":"172.21.15.165:6815/1264754008","heartbeat_front_addr":"172.21.15.165:6813/1264754008","state":["exists","up"]},{"osd":6,"uuid":"d1f9bcec-338a-44b6-bddb-920dac2a46e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6819","nonce":2663871562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6821","nonce":2663871562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6825","nonce":2663871562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2663871562},{"type":"v1","addr":"172.21.15.2:6823","nonce":2663871562}]},"public_addr":"172.21.15.2:6819/2663871562","cluster_addr":"172.21.15.2:6821/2663871562","heartbeat_back_addr":"172.21.15.2:6825/2663871562","heartbeat_front_addr":"172.21.15.2:6823/2663871562","state":["exists","up"]},{"osd":7,"uuid":"787a342e-4f3a-47f0-9ff8-cf7e80400e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6817","nonce":2299093743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6819","nonce":2299093743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6823","nonce":2299093743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2299093743},{"type":"v1","addr":"172.21.15.23:6821","nonce":2299093743}]},"public_addr":"172.21.15.23:6817/2299093743","cluster_addr":"172.21.15.23:6819/2299093743","heartbeat_back_addr":"172.21.15.23:6823/2299093743","heartbeat_front_addr":"172.21.15.23:6821/2299093743","state":["exists","up"]},{"osd":8,"uuid":"4e50d064-174a-482a-8643-898a23138c36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6817","nonce":2939503461}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6819","nonce":2939503461}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6823","nonce":2939503461}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":2939503461},{"type":"v1","addr":"172.21.15.165:6821","nonce":2939503461}]},"public_addr":"172.21.15.165:6817/2939503461","cluster_addr":"172.21.15.165:6819/2939503461","heartbeat_back_addr":"172.21.15.165:6823/2939503461","heartbeat_front_addr":"172.21.15.165:6821/2939503461","state":["exists","up"]},{"osd":9,"uuid":"1ca1d014-c271-49e5-9672-4ff1457beb85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6827","nonce":1412624036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6829","nonce":1412624036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6833","nonce":1412624036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1412624036},{"type":"v1","addr":"172.21.15.2:6831","nonce":1412624036}]},"public_addr":"172.21.15.2:6827/1412624036","cluster_addr":"172.21.15.2:6829/1412624036","heartbeat_back_addr":"172.21.15.2:6833/1412624036","heartbeat_front_addr":"172.21.15.2:6831/1412624036","state":["exists","up"]},{"osd":10,"uuid":"f4fc0a35-24cc-4b2d-bfe4-a15febc6ff54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6825","nonce":1131591289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6827","nonce":1131591289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6831","nonce":1131591289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":1131591289},{"type":"v1","addr":"172.21.15.23:6829","nonce":1131591289}]},"public_addr":"172.21.15.23:6825/1131591289","cluster_addr":"172.21.15.23:6827/1131591289","heartbeat_back_addr":"172.21.15.23:6831/1131591289","heartbeat_front_addr":"172.21.15.23:6829/1131591289","state":["exists","up"]},{"osd":11,"uuid":"50c89611-028f-40d8-a5b0-d36c76cd8800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6825","nonce":1832747584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6826","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6827","nonce":1832747584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6830","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6831","nonce":1832747584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6828","nonce":1832747584},{"type":"v1","addr":"172.21.15.165:6829","nonce":1832747584}]},"public_addr":"172.21.15.165:6825/1832747584","cluster_addr":"172.21.15.165:6827/1832747584","heartbeat_back_addr":"172.21.15.165:6831/1832747584","heartbeat_front_addr":"172.21.15.165:6829/1832747584","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:18.924946+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:24:44.788564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:08.344408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:25:42.303326+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:09.529671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:26:34.088024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:08.330455+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:27:36.019212+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:01.257294+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:28:34.871756+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:02.537971+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T18:29:27.978725+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/3303291273":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2218375343":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3512553041":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3920276264":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/247107605":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/3822870648":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/1969024355":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6800/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:6801/1779941333":"2024-03-12T18:21:53.065562+0000","172.21.15.2:6801/2981412802":"2024-03-12T18:22:28.050188+0000","172.21.15.2:0/3359009707":"2024-03-12T18:21:53.065562+0000","172.21.15.2:0/492479102":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/2651572933":"2024-03-12T18:33:42.314565+0000","172.21.15.2:6800/1446641624":"2024-03-12T18:33:42.314565+0000","172.21.15.2:0/3571811795":"2024-03-12T18:33:42.314565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T18:44:59.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.792+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 msgr2=0x7f712c0a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.792+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 0x7f712c0a40e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7148198e80 tx=0x7f7130005f90 comp rx=0 tx=0).stop 2024-03-11T18:44:59.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.792+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 msgr2=0x7f714819d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.792+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f713800e9f0 tx=0x7f713800eeb0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 shutdown_connections 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7148197630 0x7f7148197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f712c0a1c20 0x7f712c0a40e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7148106450 0x7f714819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 --2- 172.21.15.2:0/718547149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7148101380 0x7f714819d570 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 >> 172.21.15.2:0/718547149 conn(0x7f71480fcc20 msgr2=0x7f71480feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 shutdown_connections 2024-03-11T18:44:59.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:44:59.793+0000 7f714f400640 1 -- 172.21.15.2:0/718547149 wait complete. 2024-03-11T18:44:59.814 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-11T18:45:00.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 -- 172.21.15.2:0/1988784261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 msgr2=0x7f8c24114710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1988784261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c24114710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c009cd0 tx=0x7f8c1c02c0e0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 -- 172.21.15.2:0/1988784261 shutdown_connections 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1988784261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c24114710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1988784261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c241078a0 0x7f8c24107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1988784261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c24109590 0x7f8c24109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 -- 172.21.15.2:0/1988784261 >> 172.21.15.2:0/1988784261 conn(0x7f8c24077020 msgr2=0x7f8c24105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:00.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 -- 172.21.15.2:0/1988784261 shutdown_connections 2024-03-11T18:45:00.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.037+0000 7f8c2a600640 1 -- 172.21.15.2:0/1988784261 wait complete. 2024-03-11T18:45:00.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.038+0000 7f8c2a600640 1 Processor -- start 2024-03-11T18:45:00.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 -- start start 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c2419e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c24109590 0x7f8c241a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c241a2ce0 con 0x7f8c24108240 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c241a2e50 con 0x7f8c241078a0 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c2a600640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c241a3010 con 0x7f8c24109590 2024-03-11T18:45:00.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c23400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c23400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:36228/0 (socket says 172.21.15.2:36228) 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c23400640 1 -- 172.21.15.2:0/1885353916 learned_addr learned my addr 172.21.15.2:0/1885353916 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c24109590 0x7f8c241a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.039+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c2419e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 -- 172.21.15.2:0/1885353916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c24109590 msgr2=0x7f8c241a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c24109590 0x7f8c241a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 -- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 msgr2=0x7f8c2419e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c2419e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 -- 172.21.15.2:0/1885353916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c009900 con 0x7f8c241078a0 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c23400640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400e9f0 tx=0x7f8c1400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14009700 con 0x7f8c241078a0 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.040+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c241a32f0 con 0x7f8c241078a0 2024-03-11T18:45:00.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.041+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c1400cd80 con 0x7f8c241078a0 2024-03-11T18:45:00.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.041+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14018920 con 0x7f8c241078a0 2024-03-11T18:45:00.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.041+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c241aab70 con 0x7f8c241078a0 2024-03-11T18:45:00.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.042+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c24068450 con 0x7f8c241078a0 2024-03-11T18:45:00.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.044+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f8c14018a80 con 0x7f8c241078a0 2024-03-11T18:45:00.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.045+0000 7f8c1be00640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 0x7f8c080a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.046+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9576+0+0 (secure 0 0 0) 0x7f8c14014070 con 0x7f8c241078a0 2024-03-11T18:45:00.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.046+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 0x7f8c080a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.047+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 0x7f8c080a4060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c2410bd80 tx=0x7f8c10005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:00.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.050+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c14052d20 con 0x7f8c241078a0 2024-03-11T18:45:00.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:59 smithi165 ceph-mon[36634]: pgmap v353: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-03-11T18:45:00.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:59 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3230625665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:44:59 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/718547149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:59 smithi023 ceph-mon[36406]: pgmap v353: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-03-11T18:45:00.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:59 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3230625665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:44:59 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/718547149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:59 smithi002 ceph-mon[31328]: pgmap v353: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2024-03-11T18:45:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:59 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3230625665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:44:59 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/718547149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T18:45:00.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.318+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8c24117a00 con 0x7f8c241078a0 2024-03-11T18:45:00.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.331+0000 7f8c1be00640 1 -- 172.21.15.2:0/1885353916 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8c14096ae0 con 0x7f8c241078a0 2024-03-11T18:45:00.334 INFO:teuthology.orchestra.run.smithi002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-11T18:45:00.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 msgr2=0x7f8c080a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 0x7f8c080a4060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c2410bd80 tx=0x7f8c10005f90 comp rx=0 tx=0).stop 2024-03-11T18:45:00.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 msgr2=0x7f8c2419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400e9f0 tx=0x7f8c1400eeb0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 shutdown_connections 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8c24109590 0x7f8c241a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f8c080a1ba0 0x7f8c080a4060 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c24108240 0x7f8c2419e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 --2- 172.21.15.2:0/1885353916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c241078a0 0x7f8c2419de20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 >> 172.21.15.2:0/1885353916 conn(0x7f8c24077020 msgr2=0x7f8c2410ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:00.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 shutdown_connections 2024-03-11T18:45:00.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.335+0000 7f8c2a600640 1 -- 172.21.15.2:0/1885353916 wait complete. 2024-03-11T18:45:00.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 -- 172.21.15.2:0/1039515351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 msgr2=0x7f940c111a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 --2- 172.21.15.2:0/1039515351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c111a10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009900 tx=0x7f93f802bcc0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 -- 172.21.15.2:0/1039515351 shutdown_connections 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 --2- 172.21.15.2:0/1039515351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c114340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 --2- 172.21.15.2:0/1039515351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c111a10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 --2- 172.21.15.2:0/1039515351 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 0x7f940c073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 -- 172.21.15.2:0/1039515351 >> 172.21.15.2:0/1039515351 conn(0x7f940c0fcad0 msgr2=0x7f940c0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:00.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 -- 172.21.15.2:0/1039515351 shutdown_connections 2024-03-11T18:45:00.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.563+0000 7f940b200640 1 -- 172.21.15.2:0/1039515351 wait complete. 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.564+0000 7f940b200640 1 Processor -- start 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.564+0000 7f940b200640 1 -- start start 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 0x7f940c1a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c1a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c19e140 con 0x7f940c111f50 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c19e2b0 con 0x7f940c073e40 2024-03-11T18:45:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940b200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c19e470 con 0x7f940c073520 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55276/0 (socket says 172.21.15.2:55276) 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f940a800640 1 -- 172.21.15.2:0/301857928 learned_addr learned my addr 172.21.15.2:0/301857928 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f9409400640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c1a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.565+0000 7f9409e00640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 0x7f940c1a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 -- 172.21.15.2:0/301857928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 msgr2=0x7f940c1a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 0x7f940c1a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 -- 172.21.15.2:0/301857928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 msgr2=0x7f940c1a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c1a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 -- 172.21.15.2:0/301857928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8009560 con 0x7f940c111f50 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f9409400640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c1a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940a800640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f940400eb40 tx=0x7f940400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404004b20 con 0x7f940c111f50 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9404004c80 con 0x7f940c111f50 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c19e750 con 0x7f940c111f50 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040107b0 con 0x7f940c111f50 2024-03-11T18:45:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.566+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c1aa940 con 0x7f940c111f50 2024-03-11T18:45:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.568+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c04eee0 con 0x7f940c111f50 2024-03-11T18:45:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.570+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f9404005020 con 0x7f940c111f50 2024-03-11T18:45:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.571+0000 7f9402a00640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 0x7f93f00a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.571+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9746+0+0 (secure 0 0 0) 0x7f9404014070 con 0x7f940c111f50 2024-03-11T18:45:00.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.571+0000 7f9409e00640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 0x7f93f00a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:00.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.572+0000 7f9409e00640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 0x7f93f00a4060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f940c1006a0 tx=0x7f93fc005f90 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.576+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9404096dd0 con 0x7f940c111f50 2024-03-11T18:45:00.656 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mds[67373]: -- 172.21.15.165:0/1304740310 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5588b66981a0 con 0x5588b666ec00 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: starting mds.c at 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.657 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mds[67144]: -- 172.21.15.23:0/1160510285 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556fc68781a0 con 0x556fc684e800 2024-03-11T18:45:00.658 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: starting mds.e at 2024-03-11T18:45:00.658 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.658 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.658 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mds[67063]: -- 172.21.15.165:0/105784668 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5573b65ca1a0 con 0x5573b65a0c00 2024-03-11T18:45:00.658 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: starting mds.f at 2024-03-11T18:45:00.659 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.659 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.659 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mds[67456]: -- 172.21.15.23:0/1046356424 <== mon.2 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56546998a1a0 con 0x565469961000 2024-03-11T18:45:00.659 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: starting mds.h at 2024-03-11T18:45:00.659 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mds[67690]: -- 172.21.15.165:0/2381611170 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d717101a0 con 0x557d716e6800 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: starting mds.i at 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: ignoring --setuser ceph since I am not root 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: ignoring --setgroup ceph since I am not root 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mds[67763]: -- 172.21.15.23:0/3058781115 <== mon.1 v2:172.21.15.165:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f16372c1a0 con 0x55f163702c00 2024-03-11T18:45:00.660 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: starting mds.b at 2024-03-11T18:45:00.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a[31324]: 2024-03-11T18:45:00.319+0000 7f5276400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T18:45:00.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:00.820+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f940c104470 con 0x7f940c111f50 2024-03-11T18:45:01.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/1885353916' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T18:45:01.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T18:45:01.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T18:45:01.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: osdmap e109: 12 total, 12 up, 12 in 2024-03-11T18:45:01.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-11T18:45:01.004 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:00 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:45:01.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/1885353916' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T18:45:01.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T18:45:01.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: osdmap e109: 12 total, 12 up, 12 in 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:00 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/1885353916' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-11T18:45:01.157 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T18:45:01.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T18:45:01.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-11T18:45:01.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: osdmap e109: 12 total, 12 up, 12 in 2024-03-11T18:45:01.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-11T18:45:01.158 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:00 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T18:45:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.332+0000 7f9402a00640 1 -- 172.21.15.2:0/301857928 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f940409a490 con 0x7f940c111f50 2024-03-11T18:45:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 msgr2=0x7f93f00a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 0x7f93f00a4060 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f940c1006a0 tx=0x7f93fc005f90 comp rx=0 tx=0).stop 2024-03-11T18:45:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 msgr2=0x7f940c19daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f940400eb40 tx=0x7f940400c2d0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 shutdown_connections 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f93f00a1ba0 0x7f93f00a4060 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c111f50 0x7f940c19daf0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f940c073e40 0x7f940c1a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 --2- 172.21.15.2:0/301857928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f940c073520 0x7f940c1a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 >> 172.21.15.2:0/301857928 conn(0x7f940c0fcad0 msgr2=0x7f940c0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:01.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.336+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 shutdown_connections 2024-03-11T18:45:01.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.337+0000 7f940b200640 1 -- 172.21.15.2:0/301857928 wait complete. 2024-03-11T18:45:01.366 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-11T18:45:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 -- 172.21.15.2:0/905291058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 msgr2=0x7fc2a41015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 --2- 172.21.15.2:0/905291058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a41015d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009c10 tx=0x7fc2a002bee0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 -- 172.21.15.2:0/905291058 shutdown_connections 2024-03-11T18:45:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 --2- 172.21.15.2:0/905291058 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a41062c0 0x7fc2a410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 --2- 172.21.15.2:0/905291058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a4101ba0 0x7fc2a4105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 --2- 172.21.15.2:0/905291058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a41015d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 -- 172.21.15.2:0/905291058 >> 172.21.15.2:0/905291058 conn(0x7fc2a40fca90 msgr2=0x7fc2a40feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:01.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 -- 172.21.15.2:0/905291058 shutdown_connections 2024-03-11T18:45:01.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.574+0000 7fc2ab200640 1 -- 172.21.15.2:0/905291058 wait complete. 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 Processor -- start 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 -- start start 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a4199630 0x7fc2a4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a419a100 con 0x7fc2a41062c0 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a419a270 con 0x7fc2a41011f0 2024-03-11T18:45:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2ab200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a419a430 con 0x7fc2a4199630 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2a9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2a9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55290/0 (socket says 172.21.15.2:55290) 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2a9400640 1 -- 172.21.15.2:0/9275615 learned_addr learned my addr 172.21.15.2:0/9275615 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2a9e00640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.576+0000 7fc2aa800640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a4199630 0x7fc2a4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 -- 172.21.15.2:0/9275615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a4199630 msgr2=0x7fc2a4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a4199630 0x7fc2a4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 -- 172.21.15.2:0/9275615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 msgr2=0x7fc2a419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 -- 172.21.15.2:0/9275615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294009490 con 0x7fc2a41062c0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2a9400640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc29400e9a0 tx=0x7fc29400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29400cd80 con 0x7fc2a41062c0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a0009840 con 0x7fc2a41062c0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2940035f0 con 0x7fc2a41062c0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.577+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc294010690 con 0x7fc2a41062c0 2024-03-11T18:45:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.578+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a41a6770 con 0x7fc2a41062c0 2024-03-11T18:45:01.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.579+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a41026c0 con 0x7fc2a41062c0 2024-03-11T18:45:01.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.581+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fc294003ad0 con 0x7fc2a41062c0 2024-03-11T18:45:01.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.582+0000 7fc29ea00640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 0x7fc28c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:01.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.582+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9746+0+0 (secure 0 0 0) 0x7fc294014070 con 0x7fc2a41062c0 2024-03-11T18:45:01.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.582+0000 7fc2a9e00640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 0x7fc28c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:01.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.583+0000 7fc2a9e00640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 0x7fc28c0a4060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00096f0 tx=0x7fc2a0000f40 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:01.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.585+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc294096c70 con 0x7fc2a41062c0 2024-03-11T18:45:01.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:01.779+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc2a4101ba0 con 0x7fc2a41062c0 2024-03-11T18:45:02.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: pgmap v355: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2024-03-11T18:45:02.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T18:45:02.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T18:45:02.156 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: Cluster is now healthy 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6836/789425220,v1:172.21.15.23:6839/789425220] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6834/2495167259,v1:172.21.15.165:6835/2495167259] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6835/493062958,v1:172.21.15.23:6838/493062958] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.23:6834/3431948481,v1:172.21.15.23:6837/3431948481] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6836/3369316183,v1:172.21.15.165:6838/3369316183] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: mds.? [v2:172.21.15.165:6837/2210104882,v1:172.21.15.165:6839/2210104882] up:boot 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: fsmap 12 up:standby 2024-03-11T18:45:02.157 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:45:02.158 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:01 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: pgmap v355: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: Cluster is now healthy 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6836/789425220,v1:172.21.15.23:6839/789425220] up:boot 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6834/2495167259,v1:172.21.15.165:6835/2495167259] up:boot 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6835/493062958,v1:172.21.15.23:6838/493062958] up:boot 2024-03-11T18:45:02.159 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.23:6834/3431948481,v1:172.21.15.23:6837/3431948481] up:boot 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6836/3369316183,v1:172.21.15.165:6838/3369316183] up:boot 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: mds.? [v2:172.21.15.165:6837/2210104882,v1:172.21.15.165:6839/2210104882] up:boot 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: fsmap 12 up:standby 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:45:02.160 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:45:02.161 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:45:02.161 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:01 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:02.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: pgmap v355: 129 pgs: 129 active+clean; 95 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: Cluster is now healthy 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/301857928' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6836/789425220,v1:172.21.15.23:6839/789425220] up:boot 2024-03-11T18:45:02.254 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6834/2495167259,v1:172.21.15.165:6835/2495167259] up:boot 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6835/493062958,v1:172.21.15.23:6838/493062958] up:boot 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.23:6834/3431948481,v1:172.21.15.23:6837/3431948481] up:boot 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6836/3369316183,v1:172.21.15.165:6838/3369316183] up:boot 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: mds.? [v2:172.21.15.165:6837/2210104882,v1:172.21.15.165:6839/2210104882] up:boot 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: fsmap 12 up:standby 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-11T18:45:02.255 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-11T18:45:02.256 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-11T18:45:02.256 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='mgr.25418 172.21.15.2:0/2169469756' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-11T18:45:02.256 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:01 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:02.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.335+0000 7fc29ea00640 1 -- 172.21.15.2:0/9275615 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc294003760 con 0x7fc2a41062c0 2024-03-11T18:45:02.338 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' removed 2024-03-11T18:45:02.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.341+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 msgr2=0x7fc28c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:02.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.341+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 0x7fc28c0a4060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00096f0 tx=0x7fc2a0000f40 comp rx=0 tx=0).stop 2024-03-11T18:45:02.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.341+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 msgr2=0x7fc2a419fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:02.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.341+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc29400e9a0 tx=0x7fc29400eda0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 shutdown_connections 2024-03-11T18:45:02.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2a4199630 0x7fc2a4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fc28c0a1ba0 0x7fc28c0a4060 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2a41062c0 0x7fc2a419fab0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 --2- 172.21.15.2:0/9275615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2a41011f0 0x7fc2a419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.342+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 >> 172.21.15.2:0/9275615 conn(0x7fc2a40fca90 msgr2=0x7fc2a40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:02.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.344+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 shutdown_connections 2024-03-11T18:45:02.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.344+0000 7fc2ab200640 1 -- 172.21.15.2:0/9275615 wait complete. 2024-03-11T18:45:02.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 -- 172.21.15.2:0/647834030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073520 msgr2=0x7f4aa4073900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 --2- 172.21.15.2:0/647834030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073520 0x7f4aa4073900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009900 tx=0x7f4a9802bcc0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 -- 172.21.15.2:0/647834030 shutdown_connections 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 --2- 172.21.15.2:0/647834030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 0x7f4aa4114340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 --2- 172.21.15.2:0/647834030 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073e40 0x7f4aa4111a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 --2- 172.21.15.2:0/647834030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073520 0x7f4aa4073900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 -- 172.21.15.2:0/647834030 >> 172.21.15.2:0/647834030 conn(0x7f4aa40fcad0 msgr2=0x7f4aa40feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:02.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.574+0000 7f4aaa200640 1 -- 172.21.15.2:0/647834030 shutdown_connections 2024-03-11T18:45:02.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.575+0000 7f4aaa200640 1 -- 172.21.15.2:0/647834030 wait complete. 2024-03-11T18:45:02.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.576+0000 7f4aaa200640 1 Processor -- start 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.576+0000 7f4aaa200640 1 -- start start 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073520 0x7f4aa41a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 0x7f4aa419da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa419e0a0 con 0x7f4aa4111f50 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa419e210 con 0x7f4aa4073e40 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aaa200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa419e3d0 con 0x7f4aa4073520 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:02.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.2:36262/0 (socket says 172.21.15.2:36262) 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 0x7f4aa419da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 -- 172.21.15.2:0/3078256518 learned_addr learned my addr 172.21.15.2:0/3078256518 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa8e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073520 0x7f4aa41a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 -- 172.21.15.2:0/3078256518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073520 msgr2=0x7f4aa41a38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073520 0x7f4aa41a38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 -- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 msgr2=0x7f4aa419da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 0x7f4aa419da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.577+0000 7f4aa3e00640 1 -- 172.21.15.2:0/3078256518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98009560 con 0x7f4aa4073e40 2024-03-11T18:45:02.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aa3e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000e9f0 tx=0x7f4a9000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:02.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90009720 con 0x7f4aa4073e40 2024-03-11T18:45:02.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a9000cd80 con 0x7f4aa4073e40 2024-03-11T18:45:02.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa419e6b0 con 0x7f4aa4073e40 2024-03-11T18:45:02.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90010690 con 0x7f4aa4073e40 2024-03-11T18:45:02.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.578+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa41aa940 con 0x7f4aa4073e40 2024-03-11T18:45:02.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.579+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa404eee0 con 0x7f4aa4073e40 2024-03-11T18:45:02.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.582+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7f4a900046c0 con 0x7f4aa4073e40 2024-03-11T18:45:02.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.583+0000 7f4aa1600640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 0x7f4a8c0a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:02.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.583+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 9163+0+0 (secure 0 0 0) 0x7f4a90014070 con 0x7f4aa4073e40 2024-03-11T18:45:02.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.583+0000 7f4aa8e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 0x7f4a8c0a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:02.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.584+0000 7f4aa8e00640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 0x7f4a8c0a4060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005ab0 tx=0x7f4a98005a20 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:02.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.585+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4a900969f0 con 0x7f4aa4073e40 2024-03-11T18:45:02.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:02.786+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4aa41044d0 con 0x7f4aa4073e40 2024-03-11T18:45:03.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.357+0000 7f4aa1600640 1 -- 172.21.15.2:0/3078256518 <== mon.2 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4aa41044d0 con 0x7f4aa4073e40 2024-03-11T18:45:03.361 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' removed 2024-03-11T18:45:03.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 msgr2=0x7f4a8c0a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:03.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 0x7f4a8c0a4060 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005ab0 tx=0x7f4a98005a20 comp rx=0 tx=0).stop 2024-03-11T18:45:03.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 msgr2=0x7f4aa41a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:03.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000e9f0 tx=0x7f4a9000edf0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 shutdown_connections 2024-03-11T18:45:03.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7f4a8c0a1ba0 0x7f4a8c0a4060 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4aa4111f50 0x7f4aa419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4aa4073e40 0x7f4aa41a3de0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 --2- 172.21.15.2:0/3078256518 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4aa4073520 0x7f4aa41a38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.364+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 >> 172.21.15.2:0/3078256518 conn(0x7f4aa40fcad0 msgr2=0x7f4aa40fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:03.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.365+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 shutdown_connections 2024-03-11T18:45:03.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.365+0000 7f4aaa200640 1 -- 172.21.15.2:0/3078256518 wait complete. 2024-03-11T18:45:03.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 -- 172.21.15.2:0/3487566702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb348107580 msgr2=0x7fb348107a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 --2- 172.21.15.2:0/3487566702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb348107580 0x7fb348107a00 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb338009860 tx=0x7fb33802bc00 comp rx=0 tx=0).stop 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 -- 172.21.15.2:0/3487566702 shutdown_connections 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 --2- 172.21.15.2:0/3487566702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107f40 0x7fb348114430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 --2- 172.21.15.2:0/3487566702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb348107580 0x7fb348107a00 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 --2- 172.21.15.2:0/3487566702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb34810c540 0x7fb34810c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.595+0000 7fb347200640 1 -- 172.21.15.2:0/3487566702 >> 172.21.15.2:0/3487566702 conn(0x7fb348076d80 msgr2=0x7fb3480791a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:03.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.596+0000 7fb347200640 1 -- 172.21.15.2:0/3487566702 shutdown_connections 2024-03-11T18:45:03.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.596+0000 7fb347200640 1 -- 172.21.15.2:0/3487566702 wait complete. 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.596+0000 7fb347200640 1 Processor -- start 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 -- start start 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb348107f40 0x7fb3481a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb34810c540 0x7fb34819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819e0a0 con 0x7fb348107580 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819e210 con 0x7fb34810c540 2024-03-11T18:45:03.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb347200640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34819e3d0 con 0x7fb348107f40 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb345e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb345e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55316/0 (socket says 172.21.15.2:55316) 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb345e00640 1 -- 172.21.15.2:0/541471216 learned_addr learned my addr 172.21.15.2:0/541471216 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb33fe00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb348107f40 0x7fb3481a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.597+0000 7fb346800640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb34810c540 0x7fb34819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 -- 172.21.15.2:0/541471216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb348107f40 msgr2=0x7fb3481a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb348107f40 0x7fb3481a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 -- 172.21.15.2:0/541471216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb34810c540 msgr2=0x7fb34819da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb34810c540 0x7fb34819da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 -- 172.21.15.2:0/541471216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338009490 con 0x7fb348107580 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb345e00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e9c0 tx=0x7fb33000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330009700 con 0x7fb348107580 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb33000cd80 con 0x7fb348107580 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34819e6b0 con 0x7fb348107580 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.598+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330010430 con 0x7fb348107580 2024-03-11T18:45:03.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.599+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3481aa8f0 con 0x7fb348107580 2024-03-11T18:45:03.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.600+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34804eee0 con 0x7fb348107580 2024-03-11T18:45:03.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.601+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 136409+0+0 (secure 0 0 0) 0x7fb330003680 con 0x7fb348107580 2024-03-11T18:45:03.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.603+0000 7fb33ea00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 0x7fb3280a4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T18:45:03.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.603+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 8625+0+0 (secure 0 0 0) 0x7fb330014070 con 0x7fb348107580 2024-03-11T18:45:03.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.603+0000 7fb33fe00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 0x7fb3280a4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T18:45:03.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.604+0000 7fb33fe00640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 0x7fb3280a4060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb33802c210 tx=0x7fb3380056d0 comp rx=0 tx=0).ready entity=mgr.25418 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T18:45:03.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.607+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3300969d0 con 0x7fb348107580 2024-03-11T18:45:03.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:03 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:03.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:03 smithi165 ceph-mon[36634]: osdmap e110: 12 total, 12 up, 12 in 2024-03-11T18:45:03.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:03 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/3078256518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:03 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:03 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:03.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:03 smithi023 ceph-mon[36406]: osdmap e110: 12 total, 12 up, 12 in 2024-03-11T18:45:03.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:03 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/3078256518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:03 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:03 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/9275615' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:03.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:03 smithi002 ceph-mon[31328]: osdmap e110: 12 total, 12 up, 12 in 2024-03-11T18:45:03.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:03 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/3078256518' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:03 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:03.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:03.810+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb34819ef30 con 0x7fb348107580 2024-03-11T18:45:04.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.351+0000 7fb33ea00640 1 -- 172.21.15.2:0/541471216 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v112) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb34819ef30 con 0x7fb348107580 2024-03-11T18:45:04.354 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data_ec' removed 2024-03-11T18:45:04.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 msgr2=0x7fb3280a4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:04.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 0x7fb3280a4060 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb33802c210 tx=0x7fb3380056d0 comp rx=0 tx=0).stop 2024-03-11T18:45:04.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 msgr2=0x7fb3481a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T18:45:04.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e9c0 tx=0x7fb33000ee80 comp rx=0 tx=0).stop 2024-03-11T18:45:04.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 shutdown_connections 2024-03-11T18:45:04.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb34810c540 0x7fb34819da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:04.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb348107f40 0x7fb3481a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:04.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:6800/1417175579,v1:172.21.15.2:6801/1417175579] conn(0x7fb3280a1ba0 0x7fb3280a4060 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:04.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 --2- 172.21.15.2:0/541471216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb348107580 0x7fb3481a3850 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T18:45:04.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.358+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 >> 172.21.15.2:0/541471216 conn(0x7fb348076d80 msgr2=0x7fb348077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T18:45:04.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.359+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 shutdown_connections 2024-03-11T18:45:04.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-03-11T18:45:04.359+0000 7fb347200640 1 -- 172.21.15.2:0/541471216 wait complete. 2024-03-11T18:45:04.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:45:04.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:45:04.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:45:04.565 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T18:45:04.566 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T18:45:04.607 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T18:45:04.623 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T18:45:04.639 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T18:45:04.640 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T18:45:04.640 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a 2024-03-11T18:45:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:04 smithi165 ceph-mon[36634]: pgmap v357: 97 pgs: 97 active+clean; 54 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:45:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:04 smithi165 ceph-mon[36634]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:04 smithi165 ceph-mon[36634]: osdmap e111: 12 total, 12 up, 12 in 2024-03-11T18:45:04.656 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:04 smithi165 ceph-mon[36634]: from='client.? 172.21.15.2:0/541471216' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:04 smithi023 ceph-mon[36406]: pgmap v357: 97 pgs: 97 active+clean; 54 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:45:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:04 smithi023 ceph-mon[36406]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:04 smithi023 ceph-mon[36406]: osdmap e111: 12 total, 12 up, 12 in 2024-03-11T18:45:04.657 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:04 smithi023 ceph-mon[36406]: from='client.? 172.21.15.2:0/541471216' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:04.722 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:04 smithi002 ceph-mon[31328]: pgmap v357: 97 pgs: 97 active+clean; 54 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-11T18:45:04.722 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:04 smithi002 ceph-mon[31328]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T18:45:04.723 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:04 smithi002 ceph-mon[31328]: osdmap e111: 12 total, 12 up, 12 in 2024-03-11T18:45:04.723 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:04 smithi002 ceph-mon[31328]: from='client.? 172.21.15.2:0/541471216' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T18:45:05.003 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:04 smithi002 systemd[1]: Stopping Ceph mon.a for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:05.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a[31324]: 2024-03-11T18:45:05.191+0000 7f527d200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:05.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a[31324]: 2024-03-11T18:45:05.191+0000 7f527d200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T18:45:05.504 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 podman[94591]: 2024-03-11 18:45:05.23409637 +0000 UTC m=+0.179324339 container died e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T18:45:05.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 podman[94591]: 2024-03-11 18:45:05.685173455 +0000 UTC m=+0.630401414 container cleanup e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T18:45:05.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 bash[94591]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a 2024-03-11T18:45:06.253 INFO:journalctl@ceph.mon.a.smithi002.stdout:Mar 11 18:45:05 smithi002 podman[94604]: 2024-03-11 18:45:05.978152517 +0000 UTC m=+0.726962568 container remove e64cf63ba82972cd7c5dbf860584f720115bbe7272f50da3520da2babf74fc1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T18:45:06.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T18:45:06.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:06.706 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T18:45:06.706 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T18:45:06.706 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.b 2024-03-11T18:45:07.034 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:06 smithi023 systemd[1]: Stopping Ceph mon.b for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:07.406 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b[36402]: 2024-03-11T18:45:07.032+0000 7f9da07a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:07.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b[36402]: 2024-03-11T18:45:07.032+0000 7f9da07a2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T18:45:07.407 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 podman[73990]: 2024-03-11 18:45:07.236623152 +0000 UTC m=+0.328835649 container died 500d4b783fd695ca4e9c5ec4315043a822c4476db96baad82553ee2a2fa01882 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-03-11T18:45:07.795 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 podman[73990]: 2024-03-11 18:45:07.624296244 +0000 UTC m=+0.716508737 container cleanup 500d4b783fd695ca4e9c5ec4315043a822c4476db96baad82553ee2a2fa01882 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T18:45:07.795 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 bash[73990]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b 2024-03-11T18:45:08.046 INFO:journalctl@ceph.mon.b.smithi023.stdout:Mar 11 18:45:07 smithi023 podman[74005]: 2024-03-11 18:45:07.917726188 +0000 UTC m=+0.682239082 container remove 500d4b783fd695ca4e9c5ec4315043a822c4476db96baad82553ee2a2fa01882 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T18:45:08.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T18:45:08.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:08.112 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T18:45:08.112 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T18:45:08.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.c 2024-03-11T18:45:08.406 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:08 smithi165 systemd[1]: Stopping Ceph mon.c for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:08.688 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c[36614]: 2024-03-11T18:45:08.430+0000 7fef38473700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:08.688 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c[36614]: 2024-03-11T18:45:08.430+0000 7fef38473700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T18:45:09.010 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:08 smithi165 podman[74165]: 2024-03-11 18:45:08.686581188 +0000 UTC m=+0.378836363 container died 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5) 2024-03-11T18:45:09.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 podman[74165]: 2024-03-11 18:45:09.099408881 +0000 UTC m=+0.791664131 container cleanup 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:09.336 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 bash[74165]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c 2024-03-11T18:45:09.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T18:45:09.598 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 podman[74179]: 2024-03-11 18:45:09.334274499 +0000 UTC m=+0.648479568 container remove 6ee334e5e5b0c7b54aad1f5c062cdd27abfe0435dd1f5bec826c95982bf2acdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T18:45:09.598 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T18:45:09.599 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 systemd[1]: Stopped Ceph mon.c for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:45:09.599 INFO:journalctl@ceph.mon.c.smithi165.stdout:Mar 11 18:45:09 smithi165 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mon.c.service: Consumed 14.893s CPU time. 2024-03-11T18:45:09.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:09.623 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T18:45:09.623 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T18:45:09.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x 2024-03-11T18:45:10.003 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:09 smithi002 systemd[1]: Stopping Ceph mgr.x for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:10.503 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:45:10.204+0000 7faf64200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:10.503 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x[31568]: 2024-03-11T18:45:10.204+0000 7faf64200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T18:45:10.504 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 podman[94711]: 2024-03-11 18:45:10.299251275 +0000 UTC m=+0.246683330 container died b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:10.987 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 podman[94711]: 2024-03-11 18:45:10.773397762 +0000 UTC m=+0.720829811 container cleanup b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T18:45:10.987 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 bash[94711]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x 2024-03-11T18:45:11.253 INFO:journalctl@ceph.mgr.x.smithi002.stdout:Mar 11 18:45:10 smithi002 podman[94725]: 2024-03-11 18:45:10.983568787 +0000 UTC m=+0.667660948 container remove b214df16bcc6d0969e3761359a6ae098d106e45432832a63e63251fa199b72f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T18:45:11.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T18:45:11.628 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:11.628 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T18:45:11.628 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T18:45:11.628 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.y 2024-03-11T18:45:11.977 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:45:11 smithi023 systemd[1]: Stopping Ceph mgr.y for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:12.283 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:45:11 smithi023 podman[74114]: 2024-03-11 18:45:11.975174706 +0000 UTC m=+0.142370907 container died 574760bdc2eb94b4c9bcb2bd733166690eecd4f360e32ebfd2f4f6784395061f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T18:45:12.650 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:45:12 smithi023 podman[74114]: 2024-03-11 18:45:12.388700581 +0000 UTC m=+0.555896772 container cleanup 574760bdc2eb94b4c9bcb2bd733166690eecd4f360e32ebfd2f4f6784395061f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T18:45:12.650 INFO:journalctl@ceph.mgr.y.smithi023.stdout:Mar 11 18:45:12 smithi023 bash[74114]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-y 2024-03-11T18:45:12.817 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T18:45:12.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:12.859 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T18:45:12.859 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T18:45:12.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.z 2024-03-11T18:45:13.156 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:45:12 smithi165 systemd[1]: Stopping Ceph mgr.z for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:13.482 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:45:13 smithi165 podman[74286]: 2024-03-11 18:45:13.182612549 +0000 UTC m=+0.135631557 container died fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T18:45:13.906 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:45:13 smithi165 podman[74286]: 2024-03-11 18:45:13.579413318 +0000 UTC m=+0.532432327 container cleanup fd233307c8795d4338d20bf7b7241be300493ea3b55607386326ab511de5a79f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T18:45:13.906 INFO:journalctl@ceph.mgr.z.smithi165.stdout:Mar 11 18:45:13 smithi165 bash[74286]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mgr-z 2024-03-11T18:45:14.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T18:45:14.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:14.192 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T18:45:14.192 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T18:45:14.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.0 2024-03-11T18:45:14.753 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:14 smithi002 systemd[1]: Stopping Ceph osd.0 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:15.253 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0[46512]: 2024-03-11T18:45:14.842+0000 7ff3a2600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:15.254 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0[46512]: 2024-03-11T18:45:14.842+0000 7ff3a2600700 -1 osd.0 112 *** Got signal Terminated *** 2024-03-11T18:45:15.254 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0[46512]: 2024-03-11T18:45:14.842+0000 7ff3a2600700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:20.754 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:20 smithi002 podman[94830]: 2024-03-11 18:45:20.465646698 +0000 UTC m=+5.768492975 container died 6fc2dc659579265695404cf26ef5696e50dd41c1aa8132b5aaa4ecea1d85275e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T18:45:21.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:20 smithi002 podman[94830]: 2024-03-11 18:45:20.923831306 +0000 UTC m=+6.226677596 container cleanup 6fc2dc659579265695404cf26ef5696e50dd41c1aa8132b5aaa4ecea1d85275e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:21.180 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:20 smithi002 bash[94830]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0 2024-03-11T18:45:21.503 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:21 smithi002 podman[94851]: 2024-03-11 18:45:21.177496521 +0000 UTC m=+0.696484234 container remove 6fc2dc659579265695404cf26ef5696e50dd41c1aa8132b5aaa4ecea1d85275e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:45:22.402 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:21.957070401 +0000 UTC m=+0.035710225 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:45:22.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:22.084478672 +0000 UTC m=+0.163118537 container create d4f3175bea4addb63a588e85424faaec6f797ec44dc14b6e52736e6f164123a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T18:45:22.405 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:22.398958735 +0000 UTC m=+0.477598603 container init d4f3175bea4addb63a588e85424faaec6f797ec44dc14b6e52736e6f164123a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:45:22.753 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:22.410802343 +0000 UTC m=+0.489442210 container start d4f3175bea4addb63a588e85424faaec6f797ec44dc14b6e52736e6f164123a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:22.753 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:22.514705595 +0000 UTC m=+0.593345499 container attach d4f3175bea4addb63a588e85424faaec6f797ec44dc14b6e52736e6f164123a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T18:45:23.253 INFO:journalctl@ceph.osd.0.smithi002.stdout:Mar 11 18:45:22 smithi002 podman[94938]: 2024-03-11 18:45:22.860141617 +0000 UTC m=+0.938781515 container died d4f3175bea4addb63a588e85424faaec6f797ec44dc14b6e52736e6f164123a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-11T18:45:23.675 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T18:45:23.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:23.798 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T18:45:23.798 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T18:45:23.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.1 2024-03-11T18:45:24.129 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:23 smithi023 systemd[1]: Stopping Ceph osd.1 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:24.406 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1[40995]: 2024-03-11T18:45:24.126+0000 7f561c507700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:24.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1[40995]: 2024-03-11T18:45:24.126+0000 7f561c507700 -1 osd.1 112 *** Got signal Terminated *** 2024-03-11T18:45:24.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1[40995]: 2024-03-11T18:45:24.126+0000 7f561c507700 -1 osd.1 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:30.040 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:29 smithi023 podman[74238]: 2024-03-11 18:45:29.706020854 +0000 UTC m=+5.711610801 container died 0053d058a15970851e48b2ed05f32a321e4766df15748ae06fb3c0ef4540dd12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:45:30.383 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:30 smithi023 podman[74238]: 2024-03-11 18:45:30.146730181 +0000 UTC m=+6.152320134 container cleanup 0053d058a15970851e48b2ed05f32a321e4766df15748ae06fb3c0ef4540dd12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T18:45:30.383 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:30 smithi023 bash[74238]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1 2024-03-11T18:45:30.656 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:30 smithi023 podman[74257]: 2024-03-11 18:45:30.38152578 +0000 UTC m=+0.668257707 container remove 0053d058a15970851e48b2ed05f32a321e4766df15748ae06fb3c0ef4540dd12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:45:31.123 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:30 smithi023 podman[74359]: 2024-03-11 18:45:30.75721992 +0000 UTC m=+0.021857298 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:45:31.123 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:30 smithi023 podman[74359]: 2024-03-11 18:45:30.874448503 +0000 UTC m=+0.139085872 container create 36a02df98a17b31ea4aed6c83f6dfc3ec9f5f9650ddfabe3edb47bf4c62d263f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:31.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:31 smithi023 podman[74359]: 2024-03-11 18:45:31.236260001 +0000 UTC m=+0.500897387 container init 36a02df98a17b31ea4aed6c83f6dfc3ec9f5f9650ddfabe3edb47bf4c62d263f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T18:45:31.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:31 smithi023 podman[74359]: 2024-03-11 18:45:31.23991092 +0000 UTC m=+0.504548294 container start 36a02df98a17b31ea4aed6c83f6dfc3ec9f5f9650ddfabe3edb47bf4c62d263f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:31.407 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:31 smithi023 podman[74359]: 2024-03-11 18:45:31.335757521 +0000 UTC m=+0.600394910 container attach 36a02df98a17b31ea4aed6c83f6dfc3ec9f5f9650ddfabe3edb47bf4c62d263f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T18:45:31.906 INFO:journalctl@ceph.osd.1.smithi023.stdout:Mar 11 18:45:31 smithi023 podman[74359]: 2024-03-11 18:45:31.542660329 +0000 UTC m=+0.807297700 container died 36a02df98a17b31ea4aed6c83f6dfc3ec9f5f9650ddfabe3edb47bf4c62d263f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T18:45:32.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T18:45:32.298 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:32.298 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T18:45:32.298 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T18:45:32.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.2 2024-03-11T18:45:32.620 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:32 smithi165 systemd[1]: Stopping Ceph osd.2 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:32.906 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:45:32.616+0000 7f5732909700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:32.906 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:45:32.616+0000 7f5732909700 -1 osd.2 112 *** Got signal Terminated *** 2024-03-11T18:45:32.906 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2[41783]: 2024-03-11T18:45:32.616+0000 7f5732909700 -1 osd.2 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:38.520 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:38 smithi165 podman[74425]: 2024-03-11 18:45:38.212859162 +0000 UTC m=+5.723195379 container died 64a77647f7eec541555fdcc3ba99a5e1de46842003f0bfa9b42a2b2842a5ba2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T18:45:38.870 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:38 smithi165 podman[74425]: 2024-03-11 18:45:38.641870606 +0000 UTC m=+6.152206817 container cleanup 64a77647f7eec541555fdcc3ba99a5e1de46842003f0bfa9b42a2b2842a5ba2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:38.870 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:38 smithi165 bash[74425]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2 2024-03-11T18:45:39.156 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:38 smithi165 podman[74443]: 2024-03-11 18:45:38.86831362 +0000 UTC m=+0.657795750 container remove 64a77647f7eec541555fdcc3ba99a5e1de46842003f0bfa9b42a2b2842a5ba2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, org.label-schema.schema-version=1.0) 2024-03-11T18:45:39.616 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.183787775 +0000 UTC m=+0.017987007 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:45:39.616 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.311036958 +0000 UTC m=+0.145236170 container create c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T18:45:39.616 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.51389377 +0000 UTC m=+0.348092999 container init c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-11T18:45:39.616 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.516994696 +0000 UTC m=+0.351193916 container start c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T18:45:39.616 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.61367196 +0000 UTC m=+0.447871201 container attach c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T18:45:39.906 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:39 smithi165 podman[74535]: 2024-03-11 18:45:39.807069543 +0000 UTC m=+0.641268763 container died c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:45:40.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T18:45:40.364 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:40 smithi165 podman[74561]: 2024-03-11 18:45:40.317468161 +0000 UTC m=+0.501970593 container remove c4bc3b5f9c4f3ee7230a2468a1b080e05089dfd027df6eadd37f3cd7a5e726e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:40.365 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:40 smithi165 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-11T18:45:40.365 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:40 smithi165 systemd[1]: Stopped Ceph osd.2 for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:45:40.365 INFO:journalctl@ceph.osd.2.smithi165.stdout:Mar 11 18:45:40 smithi165 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.2.service: Consumed 2min 1.385s CPU time. 2024-03-11T18:45:40.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:40.382 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T18:45:40.382 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T18:45:40.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.3 2024-03-11T18:45:40.753 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:40 smithi002 systemd[1]: Stopping Ceph osd.3 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:41.137 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:40 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:40.958+0000 7efc96e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:41.138 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:40 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:40.958+0000 7efc96e00700 -1 osd.3 112 *** Got signal Terminated *** 2024-03-11T18:45:41.138 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:40 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:40.958+0000 7efc96e00700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:41.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:40.730+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:41.906 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:41.749+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:42.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:41 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:41.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:42.753 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:42.510+0000 7efc8d600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.292882+0000 front 2024-03-11T18:45:18.292855+0000 (oldest deadline 2024-03-11T18:45:41.792491+0000) 2024-03-11T18:45:43.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:42.767+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:43.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:42 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:42.870+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:43.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:42.997+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:43.834 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:43 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:43.501+0000 7efc8d600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.292882+0000 front 2024-03-11T18:45:18.292855+0000 (oldest deadline 2024-03-11T18:45:41.792491+0000) 2024-03-11T18:45:44.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:43.788+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:44.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:43 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:43.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:44.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:43.969+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:44.753 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:44.470+0000 7efc8d600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.292882+0000 front 2024-03-11T18:45:18.292855+0000 (oldest deadline 2024-03-11T18:45:41.792491+0000) 2024-03-11T18:45:45.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:44.783+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:45.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:44.735+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:45.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:44 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:44.853+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:45.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:45.002+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:45.753 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:45 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3[53481]: 2024-03-11T18:45:45.437+0000 7efc8d600700 -1 osd.3 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.292882+0000 front 2024-03-11T18:45:18.292855+0000 (oldest deadline 2024-03-11T18:45:41.792491+0000) 2024-03-11T18:45:46.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:45.776+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:46.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:45.719+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:46.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:45 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:45.854+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:46.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:45.984+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:46.838 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:46 smithi002 podman[95044]: 2024-03-11 18:45:46.520315252 +0000 UTC m=+5.690170939 container died 7c8c276b655c308fdd66355bf8d0234d5a57b7bbdfcbaccbe4be78cbd10660e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:47.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:46.748+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:47.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:46.721+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:47.207 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:46 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:46.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:47.207 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:46 smithi002 podman[95044]: 2024-03-11 18:45:46.962261188 +0000 UTC m=+6.132116872 container cleanup 7c8c276b655c308fdd66355bf8d0234d5a57b7bbdfcbaccbe4be78cbd10660e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T18:45:47.208 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:46 smithi002 bash[95044]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3 2024-03-11T18:45:47.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:46 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:46.941+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:47.503 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:47 smithi002 podman[95073]: 2024-03-11 18:45:47.205067763 +0000 UTC m=+0.676038642 container remove 7c8c276b655c308fdd66355bf8d0234d5a57b7bbdfcbaccbe4be78cbd10660e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T18:45:48.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:47.799+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:48.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:47.692+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:48.235 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:47 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:47.923+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:48.235 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:47.983052028 +0000 UTC m=+0.035951922 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:45:48.235 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:48.114429398 +0000 UTC m=+0.167329314 container create b68b2147eace23a8751c18b184867eb4825417bac7aad816641e8b685d2dc30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T18:45:48.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:47 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:47.983+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:48.503 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:48.384744429 +0000 UTC m=+0.437644389 container init b68b2147eace23a8751c18b184867eb4825417bac7aad816641e8b685d2dc30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-11T18:45:48.503 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:48.396621499 +0000 UTC m=+0.449521419 container start b68b2147eace23a8751c18b184867eb4825417bac7aad816641e8b685d2dc30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:45:48.849 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:48.50881839 +0000 UTC m=+0.561718313 container attach b68b2147eace23a8751c18b184867eb4825417bac7aad816641e8b685d2dc30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:49.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:48.818+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:49.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:48.675+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:49.194 INFO:journalctl@ceph.osd.3.smithi002.stdout:Mar 11 18:45:48 smithi002 podman[95160]: 2024-03-11 18:45:48.847571006 +0000 UTC m=+0.900470958 container died b68b2147eace23a8751c18b184867eb4825417bac7aad816641e8b685d2dc30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0) 2024-03-11T18:45:49.195 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:48 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:48.882+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:49.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:48 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:48.983+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:49.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T18:45:49.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:49.724 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T18:45:49.724 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T18:45:49.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.4 2024-03-11T18:45:50.054 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:49 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:49.987+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:50.054 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:49 smithi023 systemd[1]: Stopping Ceph osd.4 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:50.057 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:49.705+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:50.057 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:49.820+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:50.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:49 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:49.833+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:50.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:50.259+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:50.406 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:50.050+0000 7f42e006c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:50.407 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:50.050+0000 7f42e006c700 -1 osd.4 112 *** Got signal Terminated *** 2024-03-11T18:45:50.407 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:50.050+0000 7f42e006c700 -1 osd.4 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:51.018 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:50.839+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:51.019 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:50.718+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:51.020 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:50.768+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:51.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:50 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:50.821+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:51.282 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:50 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:50.951+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:51.599 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:51.279+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:51.861 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:51.763+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:51.861 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:51.682+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:51.861 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:51.682+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:51.906 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:51.596+0000 7f42dc660700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.557832+0000 front 2024-03-11T18:45:25.557857+0000 (oldest deadline 2024-03-11T18:45:51.358504+0000) 2024-03-11T18:45:52.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:51 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:51.773+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:52.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:51 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:51.561+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:52.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:51.858+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:52.290 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:51 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:51.971+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:52.592 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:52.287+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:52.906 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:52.589+0000 7f42dc660700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.557832+0000 front 2024-03-11T18:45:25.557857+0000 (oldest deadline 2024-03-11T18:45:51.358504+0000) 2024-03-11T18:45:53.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:52 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:52.599+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:53.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:52 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:52.753+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:53.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:52.824+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:53.176 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:52.824+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:53.177 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:52.773+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:53.177 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:52.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:53.177 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:52.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:53.262 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:52.929+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:53.262 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:52 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:52.929+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:53.556 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:53.259+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:53.906 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:53.554+0000 7f42dc660700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.557832+0000 front 2024-03-11T18:45:25.557857+0000 (oldest deadline 2024-03-11T18:45:51.358504+0000) 2024-03-11T18:45:54.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:53 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:53.620+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:54.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:53 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:53.705+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:54.156 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:53.742+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:54.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:53.845+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:54.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:53.845+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:54.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:53.741+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:54.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:53.741+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:54.224 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:53.959+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:54.224 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:53 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:53.959+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:54.577 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:54.221+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:54.906 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4[47966]: 2024-03-11T18:45:54.574+0000 7f42dc660700 -1 osd.4 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.557832+0000 front 2024-03-11T18:45:25.557857+0000 (oldest deadline 2024-03-11T18:45:51.358504+0000) 2024-03-11T18:45:55.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:54 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:54.574+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:55.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:54 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:54.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:55.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:54 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:54.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:45:55.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:54.866+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:55.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:54.866+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:55.156 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:54.702+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:55.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:54.772+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:55.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:54 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:54.772+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:55.268 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:54.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:55.269 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:54 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:54.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:55.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:55 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:55.265+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:55.657 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:55 smithi023 podman[74465]: 2024-03-11 18:45:55.483762126 +0000 UTC m=+5.552530021 container died dded0ce3a69bbd3f2ade3288b58f6110132c8b87cc5b4660ffaa4b91be9e5a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T18:45:56.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:55 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:55.587+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:56.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:55 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:55.746+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:56.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:55 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:55.746+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:45:56.139 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:55 smithi023 podman[74465]: 2024-03-11 18:45:55.886037212 +0000 UTC m=+5.954805098 container cleanup dded0ce3a69bbd3f2ade3288b58f6110132c8b87cc5b4660ffaa4b91be9e5a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:45:56.139 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:55 smithi023 bash[74465]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4 2024-03-11T18:45:56.140 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:55 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:55.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:56.140 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:55 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:55.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:56.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:55 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:55.818+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:56.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:55 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:55.818+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:56.156 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:55 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:55.704+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:56.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:55 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:55.774+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:56.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:55 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:55.774+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:56.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:56 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:56.309+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:56.407 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74481]: 2024-03-11 18:45:56.137351916 +0000 UTC m=+0.655015118 container remove dded0ce3a69bbd3f2ade3288b58f6110132c8b87cc5b4660ffaa4b91be9e5a26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T18:45:56.777 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74576]: 2024-03-11 18:45:56.362864055 +0000 UTC m=+0.017636577 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:45:56.777 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74576]: 2024-03-11 18:45:56.488923521 +0000 UTC m=+0.143696070 container create 1d623423194b8bf2e622d96f1c3622be28eb4b498bd410af69012a9fd06336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T18:45:56.777 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74576]: 2024-03-11 18:45:56.775085339 +0000 UTC m=+0.429857887 container init 1d623423194b8bf2e622d96f1c3622be28eb4b498bd410af69012a9fd06336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T18:45:57.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:56 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:56.549+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:57.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:56 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:56.777+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:57.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:56 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:56.777+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:45:57.089 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:57 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:56.999+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:57.089 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:57 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:56.999+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:57.089 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74576]: 2024-03-11 18:45:56.778705635 +0000 UTC m=+0.433478141 container start 1d623423194b8bf2e622d96f1c3622be28eb4b498bd410af69012a9fd06336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T18:45:57.089 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:56 smithi023 podman[74576]: 2024-03-11 18:45:56.874742888 +0000 UTC m=+0.529515412 container attach 1d623423194b8bf2e622d96f1c3622be28eb4b498bd410af69012a9fd06336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T18:45:57.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:56 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:56.816+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:57.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:56 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:56.816+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:57.157 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:56 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:56.656+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:57.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:56 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:56.773+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:57.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:56 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:56.773+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:57.355 INFO:journalctl@ceph.osd.4.smithi023.stdout:Mar 11 18:45:57 smithi023 podman[74576]: 2024-03-11 18:45:57.087525328 +0000 UTC m=+0.742297835 container died 1d623423194b8bf2e622d96f1c3622be28eb4b498bd410af69012a9fd06336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T18:45:57.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:57 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:57.353+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:57.809 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:57 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:57.535+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:57.896 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T18:45:57.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:57 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:57.621+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:57.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:57 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:57.804+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:57.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:57 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:57.804+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:57.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:57 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:57.771+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:57.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:57 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:57.771+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:57.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:45:57.942 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T18:45:57.943 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T18:45:57.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.5 2024-03-11T18:45:58.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:57 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:57.998+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:58.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:57 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:57.998+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:58.247 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 systemd[1]: Stopping Ceph osd.5 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:45:58.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:57 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:57.806+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:58.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:57 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:57.806+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:45:58.641 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:58.244+0000 7f6f3be4f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:45:58.641 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:58.244+0000 7f6f3be4f700 -1 osd.5 112 *** Got signal Terminated *** 2024-03-11T18:45:58.641 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:58.244+0000 7f6f3be4f700 -1 osd.5 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:45:58.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:58 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:58.315+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:45:58.754 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:58.546+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:45:58.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:58.633+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:45:58.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:58.764+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:45:58.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:58.764+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:45:58.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:58.803+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:45:58.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:58 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:58.803+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:45:59.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:58.786+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:45:59.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:58 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:58.786+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:45:59.335 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:59 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:59.036+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:45:59.336 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:45:59 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:59.036+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:45:59.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:45:59 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:45:59.333+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:00.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:59 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:59.562+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:00.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:45:59 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:45:59.562+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:00.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:59 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:59.797+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:00.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:45:59 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:45:59.797+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:00.156 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:45:59 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:45:59.659+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:00.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:59 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:59.783+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:46:00.156 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:45:59 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:45:59.783+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:46:00.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:59 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:59.816+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:00.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:45:59 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:45:59.816+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:00.306 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:59.998+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:00.307 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:59.998+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:00.307 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:45:59.998+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:00.624 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:00.303+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:00.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:00.627+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:00.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:00.853+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:00.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:00.853+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:00.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:00.825+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:46:00.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:00 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:00.825+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:46:01.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:00.556+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:01.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:00.556+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:01.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:00.748+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:01.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:00.748+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:01.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:00.748+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:01.258 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:00.971+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:01.259 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:00.971+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:01.259 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:00 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:00.971+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:01.595 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:01 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:01.254+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:01.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:01 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:01.635+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:01.906 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:01 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:01.787+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:46:01.906 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:01 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:01.787+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:46:01.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:01 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:01.854+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:01.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:01 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:01.854+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:02.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:01.583+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:02.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:01.583+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:02.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:01.730+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:02.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:01.730+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:02.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:01 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:01.730+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:02.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:02 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:02.208+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:02.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:02 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:02.208+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:02.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:01 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:01.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:02.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:01 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:01.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:02.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:01 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:01.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:02.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:02.623+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:02.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:02.623+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:02.799+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.173002+0000 front 2024-03-11T18:45:15.172899+0000 (oldest deadline 2024-03-11T18:45:40.472911+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:02.799+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.575516+0000 front 2024-03-11T18:45:28.575568+0000 (oldest deadline 2024-03-11T18:45:52.075610+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5[48305]: 2024-03-11T18:46:02.799+0000 7f6f38443700 -1 osd.5 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.177802+0000 front 2024-03-11T18:45:37.177823+0000 (oldest deadline 2024-03-11T18:46:01.877990+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:02.844+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:02.844+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:02.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:02 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:02.844+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:03.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:02 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:02.558+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:03.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:02 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:02.558+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:03.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:02 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:02.706+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:03.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:02 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:02.706+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:03.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:02 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:02.706+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:03.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:03.196+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:03.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:03.196+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:03.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:02 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:02.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:03.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:02 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:02.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:03.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:02 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:02.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:03.907 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:03 smithi165 podman[74653]: 2024-03-11 18:46:03.762536807 +0000 UTC m=+5.618144330 container died 24b24c985734bff7ec81f25da39796ee7455aa247287268e26e9d21f528f8fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:46:03.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:03 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:03.638+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:03.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:03 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:03.638+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:03.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:03 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:03.883+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:03.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:03 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:03.883+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:03.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:03 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:03.883+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:04.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:03 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:03.578+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:04.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:03 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:03.578+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:04.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:03 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:03.689+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:04.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:03 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:03.689+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:04.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:03 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:03.689+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:04.406 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74653]: 2024-03-11 18:46:04.160977644 +0000 UTC m=+6.016585166 container cleanup 24b24c985734bff7ec81f25da39796ee7455aa247287268e26e9d21f528f8fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:46:04.406 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 bash[74653]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5 2024-03-11T18:46:04.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:03.953+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:04.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:03.953+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:04.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:03 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:03.953+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:04.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:04 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:04.163+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:04.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:04 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:04.163+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:04.710 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74670]: 2024-03-11 18:46:04.404005903 +0000 UTC m=+0.638122238 container remove 24b24c985734bff7ec81f25da39796ee7455aa247287268e26e9d21f528f8fd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-11T18:46:04.710 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:04.592+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:04.711 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:04.592+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:04.962 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:04.878+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:04.878+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:04 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:04.878+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74763]: 2024-03-11 18:46:04.611232781 +0000 UTC m=+0.018038065 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74763]: 2024-03-11 18:46:04.729848637 +0000 UTC m=+0.136653950 container create 01f9bb75aaf3e4fb37b2d3302c41f43492d253738142d069443284733f8ab03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5-deactivate, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74763]: 2024-03-11 18:46:04.957312141 +0000 UTC m=+0.364117405 container init 01f9bb75aaf3e4fb37b2d3302c41f43492d253738142d069443284733f8ab03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T18:46:04.963 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:04 smithi165 podman[74763]: 2024-03-11 18:46:04.960913299 +0000 UTC m=+0.367718577 container start 01f9bb75aaf3e4fb37b2d3302c41f43492d253738142d069443284733f8ab03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T18:46:05.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:04 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:04.607+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:05.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:04 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:04.607+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:05.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:04 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:04.728+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:05.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:04 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:04.728+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:05.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:04 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:04.728+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:05.252 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:05 smithi165 podman[74763]: 2024-03-11 18:46:05.084482341 +0000 UTC m=+0.491287631 container attach 01f9bb75aaf3e4fb37b2d3302c41f43492d253738142d069443284733f8ab03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T18:46:05.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:05 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:05.171+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:05.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:05 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:05.171+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:05.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:04 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:04.982+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:05.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:04 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:04.982+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:05.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:04 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:04.982+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:05.599 INFO:journalctl@ceph.osd.5.smithi165.stdout:Mar 11 18:46:05 smithi165 podman[74763]: 2024-03-11 18:46:05.25078302 +0000 UTC m=+0.657588292 container died 01f9bb75aaf3e4fb37b2d3302c41f43492d253738142d069443284733f8ab03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-11T18:46:05.887 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:05 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:05.596+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:05.888 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:05 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:05.596+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:05.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T18:46:05.999 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:06.000 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T18:46:06.000 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T18:46:06.000 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.6 2024-03-11T18:46:06.008 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:05.638+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:06.009 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:05.638+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:06.009 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:05.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:06.009 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:05.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:06.009 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:05 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:05.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:06.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:05 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:05.884+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:06.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:05 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:05.884+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:06.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:05 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:05.884+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:06.217 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:05 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:05.957+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:06.218 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:05 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:05.957+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:06.218 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:05 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:05.957+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:06.503 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 systemd[1]: Stopping Ceph osd.6 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:06.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:06 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:06.215+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:06.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:06 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:06.215+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:06.870 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:06 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:06.556+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:06.870 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:06 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:06.556+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:07.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:06.772+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:07.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:06.772+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:07.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:06.772+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:07.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:06.605+0000 7fd261200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:07.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:06.605+0000 7fd261200700 -1 osd.6 112 *** Got signal Terminated *** 2024-03-11T18:46:07.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:06.605+0000 7fd261200700 -1 osd.6 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:07.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:06.620+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:07.005 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:06.621+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:07.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:06 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:06.867+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:07.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:06 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:06.867+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:07.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:06 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:06.867+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:07.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:06 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:06.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:07.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:06 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:06.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:07.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:06 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:06.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:07.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:07.170+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:07.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:07.170+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:07.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:07.170+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:07.889 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:07.551+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:07.890 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:07.551+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:07.890 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:07.551+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:08.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:07.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:08.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:07.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:08.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:07.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:08.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:07.666+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:08.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:07.666+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:08.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:07 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:07.666+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:43.468276+0000 front 2024-03-11T18:45:43.468518+0000 (oldest deadline 2024-03-11T18:46:06.968851+0000) 2024-03-11T18:46:08.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:07.887+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:08.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:07.887+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:08.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:07 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:07.887+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:08.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:07.940+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:08.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:07.940+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:08.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:07 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:07.940+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:08.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:08.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:08.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:08.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:08.408 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:08.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:08.866 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:08.546+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:08.867 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:08.546+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:08.867 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:08.546+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:09.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:08.681+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:09.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:08.681+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:09.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:08.681+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:43.468276+0000 front 2024-03-11T18:45:43.468518+0000 (oldest deadline 2024-03-11T18:46:06.968851+0000) 2024-03-11T18:46:09.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:08.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:09.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:08.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:09.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:08 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:08.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:09.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:08.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:09.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:08.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:09.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:08 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:08.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:09.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:08.975+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:09.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:08.975+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:09.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:08 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:08.975+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:09.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:09.150+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:09.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:09.150+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:09.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:09.150+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:09.867 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:09.585+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:09.867 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:09.585+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:09.867 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:09.585+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:10.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:09.674+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:10.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:09.674+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:10.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:09.674+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:43.468276+0000 front 2024-03-11T18:45:43.468518+0000 (oldest deadline 2024-03-11T18:46:06.968851+0000) 2024-03-11T18:46:10.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:09.792+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:10.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:09.792+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:10.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:09.792+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:10.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:09 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:09.792+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:10.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:09.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:10.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:09.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:10.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:09 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:09.864+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:10.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:10.197+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:10.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:10.197+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:10.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:10.197+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:10.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:09.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:10.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:09.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:10.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:09.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:10.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:09 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:09.994+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:10.831 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:10.570+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:10.831 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:10.570+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:10.831 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:10.570+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:11.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:10.656+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.662587+0000 front 2024-03-11T18:45:27.662070+0000 (oldest deadline 2024-03-11T18:45:51.162217+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:10.656+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:32.865174+0000 front 2024-03-11T18:45:32.865334+0000 (oldest deadline 2024-03-11T18:45:58.765446+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6[60200]: 2024-03-11T18:46:10.656+0000 7fd25ca00700 -1 osd.6 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:43.468276+0000 front 2024-03-11T18:45:43.468518+0000 (oldest deadline 2024-03-11T18:46:06.968851+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:10.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:10.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:10.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:11.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:10 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:10.775+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:11.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:10.828+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:11.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:10.828+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:11.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:10 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:10.828+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:11.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:11.157+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:11.157+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:11.157+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:10.961+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:10.961+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:10.961+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:11.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:10 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:10.961+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:11.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:11.799+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:11.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:11.799+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:11.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:11.799+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:11.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:11.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:11.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:11.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:11.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:11 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:11.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:12.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:11.740+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:12.003 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:11.740+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:12.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:11.740+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:12.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:11.740+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:12.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:11.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:11.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:11.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:11 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:11.993+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:12.136+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:12.136+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:12.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:12.136+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:12.478 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:12 smithi002 podman[95265]: 2024-03-11 18:46:12.223429936 +0000 UTC m=+5.764478526 container died 23b81036e154dc113f9d00ef19787037a566a85b1bbe45567f6d72964e134014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:46:12.747 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:12 smithi002 podman[95265]: 2024-03-11 18:46:12.572645811 +0000 UTC m=+6.113694395 container cleanup 23b81036e154dc113f9d00ef19787037a566a85b1bbe45567f6d72964e134014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T18:46:12.748 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:12 smithi002 bash[95265]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6 2024-03-11T18:46:12.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:12.821+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:12.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:12.821+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:12.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:12.821+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:12.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:12.574+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:12.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:12.574+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:12.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:12 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:12.574+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:13.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:12 smithi002 podman[95286]: 2024-03-11 18:46:12.78339106 +0000 UTC m=+0.549675286 container remove 23b81036e154dc113f9d00ef19787037a566a85b1bbe45567f6d72964e134014 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T18:46:13.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:12 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:12.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:13.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:12 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:12.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:13.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:12 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:12.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:13.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:12 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:12.744+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:13.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:13 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:13.167+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:13 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:13.167+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:13 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:13.167+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:12.996+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:12.996+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:12.996+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:13.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:12 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:12.996+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:13.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:13.815+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:13.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:13.815+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:13.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:13.815+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:13.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:13.588+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:13.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:13.588+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:13.908 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:13 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:13.588+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:14.003 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:13 smithi002 podman[95374]: 2024-03-11 18:46:13.562397973 +0000 UTC m=+0.035672112 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:14.004 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:13 smithi002 podman[95374]: 2024-03-11 18:46:13.699815404 +0000 UTC m=+0.173089561 container create 73b961d46440fa5c58bf6beca0be5cde69559f4a17860be391cffb8819c0b3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397) 2024-03-11T18:46:14.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:13 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:13.770+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:14.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:13 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:13.770+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:14.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:13 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:13.770+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:14.004 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:13 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:13.770+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:14.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:14.184+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:14.184+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:14.184+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.017+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.017+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.017+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:14.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.017+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:14.478 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:14 smithi002 podman[95374]: 2024-03-11 18:46:14.012370258 +0000 UTC m=+0.485644422 container init 73b961d46440fa5c58bf6beca0be5cde69559f4a17860be391cffb8819c0b3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-11T18:46:14.479 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:14 smithi002 podman[95374]: 2024-03-11 18:46:14.023688598 +0000 UTC m=+0.496962754 container start 73b961d46440fa5c58bf6beca0be5cde69559f4a17860be391cffb8819c0b3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T18:46:14.479 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:14 smithi002 podman[95374]: 2024-03-11 18:46:14.235761606 +0000 UTC m=+0.709035803 container attach 73b961d46440fa5c58bf6beca0be5cde69559f4a17860be391cffb8819c0b3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T18:46:14.753 INFO:journalctl@ceph.osd.6.smithi002.stdout:Mar 11 18:46:14 smithi002 podman[95374]: 2024-03-11 18:46:14.476541813 +0000 UTC m=+0.949816024 container died 73b961d46440fa5c58bf6beca0be5cde69559f4a17860be391cffb8819c0b3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T18:46:14.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:14.837+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:14.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:14.837+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:14.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:14.837+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:14.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:14.609+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:14.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:14.609+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:14.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:14 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:14.609+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:15.150 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:14.787+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:15.151 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:14.787+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:15.151 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:14.787+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:15.151 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:14 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:14.787+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:15.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T18:46:15.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:15.321 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T18:46:15.321 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T18:46:15.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.7 2024-03-11T18:46:15.356 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.229+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:15.356 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.229+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:15.357 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.229+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:15.357 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.997+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:15.357 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.997+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:15.357 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.997+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:15.357 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:14 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:14.997+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:15.656 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 systemd[1]: Stopping Ceph osd.7 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:15.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:15.819+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:15.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:15.819+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:15.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:15.819+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:15.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:15.593+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:15.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:15.593+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:15.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:15 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:15.593+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:16.023 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.664+0000 7fa4e7075700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:16.023 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.664+0000 7fa4e7075700 -1 osd.7 112 *** Got signal Terminated *** 2024-03-11T18:46:16.023 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:15 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:15.664+0000 7fa4e7075700 -1 osd.7 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:16.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:15 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:15.790+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:16.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:15 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:15.790+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:16.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:15 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:15.790+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:16.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:15 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:15.790+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:16.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:16.020+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:16.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:16.020+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:16.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:16.020+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:16.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:16.020+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:16.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:16.227+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:16.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:16.228+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:16.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:16.228+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:16.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:16 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:16.228+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:50.813645+0000 front 2024-03-11T18:45:50.813536+0000 (oldest deadline 2024-03-11T18:46:15.513814+0000) 2024-03-11T18:46:16.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:16.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:16.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:16.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:16.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:16.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:16.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:16.800+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:16.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:16.800+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:16.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:16 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:16.800+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:17.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:16 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:16.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:17.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:16 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:16.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:17.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:16 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:16.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:17.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:16 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:16.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:17.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:17.203+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:17.203+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:17.203+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:17.203+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:50.813645+0000 front 2024-03-11T18:45:50.813536+0000 (oldest deadline 2024-03-11T18:46:15.513814+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.015+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.015+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.015+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:17.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.015+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:17.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.015+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:17.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:17.534+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:17.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:17.534+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:17.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:17.534+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:17.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:17.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:17.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:17.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:17.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:17.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:17.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:17 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:17.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:18.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:17 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:17.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:18.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:17 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:17.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:18.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:17 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:17.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:18.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:17 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:17.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:18.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:17 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:17.796+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:18.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:18.174+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:18.406 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:18.174+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:18.174+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:18.174+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:50.813645+0000 front 2024-03-11T18:45:50.813536+0000 (oldest deadline 2024-03-11T18:46:15.513814+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.976+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.976+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.976+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.976+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:18.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:17.976+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:18.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:18.537+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:18.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:18.537+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:18.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:18.537+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:18.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:18.770+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:18.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:18.770+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:18.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:18.770+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:18.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:18 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:18.770+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:19.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:18.838+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:19.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:18.838+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:19.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:18.838+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:19.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:18.838+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:19.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:18 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:18.838+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:19.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:19.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:19.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:19.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:19.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:19.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:19.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:19.173+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:50.813645+0000 front 2024-03-11T18:45:50.813536+0000 (oldest deadline 2024-03-11T18:46:15.513814+0000) 2024-03-11T18:46:19.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:18.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:19.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:18.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:19.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:18.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:19.408 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:18.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:19.409 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:18 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:18.970+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:19.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:19.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:19.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:19.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:19.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:19.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:19.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:19.731+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:19.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:19.731+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:19.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:19.731+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:19.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:19 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:19.731+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:20.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:19.795+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:20.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:19.795+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:20.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:19.795+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:20.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:19.795+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:20.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:19 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:19.795+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:20.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:19.985+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:20.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:19.985+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:19.985+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:19.985+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:19 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:19.985+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:20.133+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.808914+0000 front 2024-03-11T18:45:25.808942+0000 (oldest deadline 2024-03-11T18:45:49.909026+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:20.133+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.411236+0000 front 2024-03-11T18:45:37.411127+0000 (oldest deadline 2024-03-11T18:46:01.511315+0000) 2024-03-11T18:46:20.407 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:20.133+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.412856+0000 front 2024-03-11T18:45:44.412874+0000 (oldest deadline 2024-03-11T18:46:06.712610+0000) 2024-03-11T18:46:20.408 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7[54982]: 2024-03-11T18:46:20.133+0000 7fa4e3669700 -1 osd.7 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:50.813645+0000 front 2024-03-11T18:45:50.813536+0000 (oldest deadline 2024-03-11T18:46:15.513814+0000) 2024-03-11T18:46:20.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:20.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:20.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:20.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:20.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:20.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:20.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:20.580+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:20.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:20.750+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:20.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:20.750+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:20.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:20.750+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:20.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:20 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:20.750+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:21.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:20.798+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:21.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:20.798+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:21.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:20.798+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:21.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:20.798+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:21.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:20 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:20.798+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:21.256 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:20.974+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:21.256 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:20.974+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:21.256 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:20.974+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:21.256 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:20.974+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:21.256 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:20 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:20.974+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:21.551 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:21 smithi023 podman[74691]: 2024-03-11 18:46:21.253788511 +0000 UTC m=+5.730523085 container died bc3dbf72b6bcf3586518be4bd284eedf7ca99a4a8ebec217e3cad13ae5f3f6b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T18:46:21.906 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:21 smithi023 podman[74691]: 2024-03-11 18:46:21.648454079 +0000 UTC m=+6.125188647 container cleanup bc3dbf72b6bcf3586518be4bd284eedf7ca99a4a8ebec217e3cad13ae5f3f6b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T18:46:21.906 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:21 smithi023 bash[74691]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7 2024-03-11T18:46:21.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:21.614+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:21.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:21.614+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:21.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:21.614+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:21.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:21.614+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:21.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:21.761+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:21.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:21.761+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:21.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:21.761+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:21.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:21 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:21.761+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:22.228 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:21 smithi023 podman[74710]: 2024-03-11 18:46:21.907877259 +0000 UTC m=+0.651187240 container remove bc3dbf72b6bcf3586518be4bd284eedf7ca99a4a8ebec217e3cad13ae5f3f6b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T18:46:22.229 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:21.995+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:22.229 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:21.995+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:22.229 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:21.995+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:22.229 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:21.995+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:22.229 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:21.995+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:22.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:21.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:22.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:21.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:22.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:21.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:22.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:21.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:22.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:21 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:21.819+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:22.513 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.129814518 +0000 UTC m=+0.018109064 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:22.513 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.289752793 +0000 UTC m=+0.178047327 container create 1022e5c02326b79e933dc853e021c4321e9bdc1ecb10bc7f2d30c1ab48b67d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T18:46:22.513 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.511471046 +0000 UTC m=+0.399765589 container init 1022e5c02326b79e933dc853e021c4321e9bdc1ecb10bc7f2d30c1ab48b67d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:46:22.813 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.514943337 +0000 UTC m=+0.403237873 container start 1022e5c02326b79e933dc853e021c4321e9bdc1ecb10bc7f2d30c1ab48b67d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T18:46:22.813 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.603253597 +0000 UTC m=+0.491548148 container attach 1022e5c02326b79e933dc853e021c4321e9bdc1ecb10bc7f2d30c1ab48b67d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:46:22.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:22.635+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:22.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:22.635+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:22.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:22.635+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:22.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:22.635+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:22.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:22.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:22.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:22.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:22.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:22.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:22.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:22 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:22.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:23.108 INFO:journalctl@ceph.osd.7.smithi023.stdout:Mar 11 18:46:22 smithi023 podman[74799]: 2024-03-11 18:46:22.811394287 +0000 UTC m=+0.699688837 container died 1022e5c02326b79e933dc853e021c4321e9bdc1ecb10bc7f2d30c1ab48b67d98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T18:46:23.108 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:22 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:22.991+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:23.108 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:22 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:22.991+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:23.109 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:22 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:22.991+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:23.109 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:22 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:22.991+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:23.109 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:22 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:22.991+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:23.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:22.812+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:23.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:22.812+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:23.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:22.812+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:23.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:22.812+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:23.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:22 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:22.812+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:23.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T18:46:23.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:23.534 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T18:46:23.535 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-11T18:46:23.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.8 2024-03-11T18:46:23.868 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:23.728+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:23.868 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:23.728+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:23.728+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:23.728+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 systemd[1]: Stopping Ceph osd.8 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.644+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.644+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.644+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:23.869 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.644+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:23.870 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.865+0000 7fccc9fa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:23.870 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.865+0000 7fccc9fa4700 -1 osd.8 112 *** Got signal Terminated *** 2024-03-11T18:46:23.870 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:23 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:23.865+0000 7fccc9fa4700 -1 osd.8 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:24.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:23.810+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:24.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:23.810+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:24.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:23.810+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:24.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:23.810+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:24.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:23 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:23.810+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:24.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:23 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:23.962+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:24.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:23 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:23.962+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:24.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:23 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:23.962+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:24.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:23 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:23.962+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:24.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:23 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:23.962+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:24.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:24.615+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:24.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:24.615+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:24.615+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:24.615+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:24.683+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:24.683+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:24.683+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:24.683+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:24.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:24 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:24.683+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:25.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:24.842+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:25.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:24.842+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:25.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:24.842+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:25.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:24.842+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:25.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:24 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:24.842+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:25.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:24.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:25.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:24.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:25.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:24.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:25.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:24.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:25.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:24 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:24.965+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:25.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:25.575+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:25.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:25.575+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:25.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:25.575+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:25.575+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:25.701+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:25.701+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:25.701+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:25.701+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:25.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:25 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:25.701+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:26.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:25.850+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:26.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:25.850+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:26.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:25.850+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:26.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:25.850+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:26.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:25 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:25.850+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:26.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:25.948+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:26.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:25.948+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:26.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:25.948+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:26.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:25.948+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:26.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:25.948+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:26.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:26.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:26.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:26.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:26.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:26.597+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:26.727+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:26.727+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:26.727+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:26.727+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:26.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:26 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:26.727+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:27.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:27.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:27.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:27.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:27.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:27.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:26 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:26.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:27.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:26 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:26.913+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:27.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:26 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:26.913+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:27.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:26 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:26.913+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:27.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:26 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:26.913+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:27.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:26 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:26.913+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:27.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:27.561+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:27.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:27.561+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:27.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:27.561+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:27.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:27.561+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:27.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:27.694+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:27.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:27.694+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:27.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:27.694+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:27.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:27.694+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:27.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:27 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:27.694+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:28.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:28.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:28.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:28.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:28.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:28.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:27 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:27.804+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:28.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:28.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:28.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:28.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:28.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:28.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:27 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:27.906+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:28.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:28.533+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.528724+0000 front 2024-03-11T18:45:28.528562+0000 (oldest deadline 2024-03-11T18:45:50.228716+0000) 2024-03-11T18:46:28.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:28.533+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:37.230207+0000 front 2024-03-11T18:45:37.230184+0000 (oldest deadline 2024-03-11T18:46:01.930324+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:28.533+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:41.931133+0000 front 2024-03-11T18:45:41.930937+0000 (oldest deadline 2024-03-11T18:46:07.230833+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8[55337]: 2024-03-11T18:46:28.533+0000 7fccc6598700 -1 osd.8 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.033782+0000 front 2024-03-11T18:45:54.033638+0000 (oldest deadline 2024-03-11T18:46:19.834394+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:28.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:28.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:28.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:28.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:28.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:28.908 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:28 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:28.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:29.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:29.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:29.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:29.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:29.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:29.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:28 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:28.841+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:29.307 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:29.308 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:29.308 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:29.308 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:29.308 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:29.308 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:28 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:28.926+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:29.503 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:29 smithi165 podman[74881]: 2024-03-11 18:46:29.24729306 +0000 UTC m=+5.519560801 container died 2897ef1d0fbb85394b1d48101fbb09d6d8c8c430a0e13a204addf21994210093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T18:46:29.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:29 smithi165 podman[74881]: 2024-03-11 18:46:29.599597918 +0000 UTC m=+5.871865658 container cleanup 2897ef1d0fbb85394b1d48101fbb09d6d8c8c430a0e13a204addf21994210093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:46:29.906 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:29 smithi165 bash[74881]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8 2024-03-11T18:46:29.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:29 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:29.707+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:29.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:29 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:29.707+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:29.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:29 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:29.707+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:29.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:29 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:29.707+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:29.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:29 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:29.707+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:30.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:30.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:30.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:30.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:30.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:30.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:29 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:29.879+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:30.260 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:29 smithi165 podman[74898]: 2024-03-11 18:46:29.917544442 +0000 UTC m=+0.672294039 container remove 2897ef1d0fbb85394b1d48101fbb09d6d8c8c430a0e13a204addf21994210093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T18:46:30.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:30.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:30.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:30.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:30.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:30.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:29.908+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:30.523 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.161575105 +0000 UTC m=+0.018767029 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:30.523 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.285395705 +0000 UTC m=+0.142587655 container create 549c3b01cbf63c4d7fbf765060dfb741a3f991ba5ca3e7e947b0bcec608d864a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T18:46:30.523 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.521314446 +0000 UTC m=+0.378506358 container init 549c3b01cbf63c4d7fbf765060dfb741a3f991ba5ca3e7e947b0bcec608d864a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux ) 2024-03-11T18:46:30.815 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.524722102 +0000 UTC m=+0.381914019 container start 549c3b01cbf63c4d7fbf765060dfb741a3f991ba5ca3e7e947b0bcec608d864a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.655240688 +0000 UTC m=+0.512432615 container attach 549c3b01cbf63c4d7fbf765060dfb741a3f991ba5ca3e7e947b0bcec608d864a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:30 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:30.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:30 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:30.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:30 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:30.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:30 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:30.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:30.816 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:30 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:30.751+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:31.152 INFO:journalctl@ceph.osd.8.smithi165.stdout:Mar 11 18:46:30 smithi165 podman[74988]: 2024-03-11 18:46:30.814246418 +0000 UTC m=+0.671438341 container died 549c3b01cbf63c4d7fbf765060dfb741a3f991ba5ca3e7e947b0bcec608d864a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-8-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T18:46:31.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:31.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:31.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:31.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:31.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:31.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:30 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:30.862+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:31.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:31.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:31.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:31.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:31.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:31.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:30 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:30.832+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:31.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.8.service' 2024-03-11T18:46:31.475 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:31.475 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-11T18:46:31.476 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-11T18:46:31.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.9 2024-03-11T18:46:31.878 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 systemd[1]: Stopping Ceph osd.9 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:32.078 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:31 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:31.777+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:32.078 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:31 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:31.777+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:32.078 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:31 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:31.777+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:32.078 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:31 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:31.777+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:32.078 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:31 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:31.777+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:32.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.816+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:32.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.816+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:32.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.816+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:32.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.817+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:32.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.817+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:32.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:31 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:31.817+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:32.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:32.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:32.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:32.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:32.255 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:32.255 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:31 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:31.874+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:32.255 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.071+0000 7f35af800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:32.255 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.071+0000 7f35af800700 -1 osd.9 112 *** Got signal Terminated *** 2024-03-11T18:46:32.255 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.071+0000 7f35af800700 -1 osd.9 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:33.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:32.754+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:33.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:32.754+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:33.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:32.754+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:33.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:32.754+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:32 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:32.754+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:33.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:33.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:33.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:32 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:32.865+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:33.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:33.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:33.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:33.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:33.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:33.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:32 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:32.924+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:34.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:34.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:34.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:34.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:34.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:34.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:33.723+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:34.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:34.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:33 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:33.830+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:34.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:34.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:34.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:34.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:34.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:34.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:33 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:33.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:35.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:35.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:34 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:34.745+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:35.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:35.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:35.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:35.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:35.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:35.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:34 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:34.872+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:35.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:35.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:35.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:35.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:35.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:35.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:34 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:34.913+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:36.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:36.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:36.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:36.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:36.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:36.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:35 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:35.752+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:36.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:36.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:36.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:36.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:36.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:36.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:36.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:35 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:35.847+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:36.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:36.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:36.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:36.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:36.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:36.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:35 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:35.904+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:37.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:37.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:37.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:37.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:37.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:37.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:36 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:36.755+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:37.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:37.159 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:36 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:36.893+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:37.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.932+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:15.599717+0000 front 2024-03-11T18:45:15.598980+0000 (oldest deadline 2024-03-11T18:45:40.899071+0000) 2024-03-11T18:46:37.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:28.405587+0000 front 2024-03-11T18:45:28.405649+0000 (oldest deadline 2024-03-11T18:45:54.305852+0000) 2024-03-11T18:46:37.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:34.307452+0000 front 2024-03-11T18:45:34.307374+0000 (oldest deadline 2024-03-11T18:46:00.207611+0000) 2024-03-11T18:46:37.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.811394+0000 front 2024-03-11T18:45:44.811504+0000 (oldest deadline 2024-03-11T18:46:09.510973+0000) 2024-03-11T18:46:37.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:53.415396+0000 front 2024-03-11T18:45:53.415450+0000 (oldest deadline 2024-03-11T18:46:16.915574+0000) 2024-03-11T18:46:37.254 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:36 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9[67076]: 2024-03-11T18:46:36.933+0000 7f35a6000700 -1 osd.9 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.522322+0000 front 2024-03-11T18:46:02.522394+0000 (oldest deadline 2024-03-11T18:46:26.622534+0000) 2024-03-11T18:46:38.056 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:37 smithi002 podman[95478]: 2024-03-11 18:46:37.788957099 +0000 UTC m=+5.863269460 container died d6eb8a798eed4257ffaf6d0621573d8fccd6d6d913021c48a947c4d97e28c681 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True) 2024-03-11T18:46:38.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:38.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:38.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:38.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:38.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:38.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:37.706+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:38.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:38.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:38.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:38.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:38.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:38.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:38.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:37 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:37.880+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:38.368 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:38 smithi002 podman[95478]: 2024-03-11 18:46:38.165368435 +0000 UTC m=+6.239680833 container cleanup d6eb8a798eed4257ffaf6d0621573d8fccd6d6d913021c48a947c4d97e28c681 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T18:46:38.369 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:38 smithi002 bash[95478]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9 2024-03-11T18:46:38.753 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:38 smithi002 podman[95496]: 2024-03-11 18:46:38.366228432 +0000 UTC m=+0.567936525 container remove d6eb8a798eed4257ffaf6d0621573d8fccd6d6d913021c48a947c4d97e28c681 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9, RELEASE=squid-94a5397, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T18:46:39.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:39.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:39.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:39.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:39.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:39.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:38 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:38.676+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:39.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:39.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:38 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:38.846+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:39.482 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.184102847 +0000 UTC m=+0.130704731 container create bac1e6e37fca88c2507302b47a7e51a0048669e88a4e9d00bf8e900e5453127f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T18:46:39.482 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.091545129 +0000 UTC m=+0.038146987 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:39.482 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.479485893 +0000 UTC m=+0.426087783 container init bac1e6e37fca88c2507302b47a7e51a0048669e88a4e9d00bf8e900e5453127f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T18:46:39.753 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.490688753 +0000 UTC m=+0.437290603 container start bac1e6e37fca88c2507302b47a7e51a0048669e88a4e9d00bf8e900e5453127f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:46:39.753 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.577839688 +0000 UTC m=+0.524441619 container attach bac1e6e37fca88c2507302b47a7e51a0048669e88a4e9d00bf8e900e5453127f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T18:46:40.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:40.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:40.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:40.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:40.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:40.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:39 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:39.678+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:40.157 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:40.158 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:39 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:39.839+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:40.253 INFO:journalctl@ceph.osd.9.smithi002.stdout:Mar 11 18:46:39 smithi002 podman[95585]: 2024-03-11 18:46:39.920617857 +0000 UTC m=+0.867219721 container died bac1e6e37fca88c2507302b47a7e51a0048669e88a4e9d00bf8e900e5453127f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:46:40.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.9.service' 2024-03-11T18:46:40.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:40.725 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-11T18:46:40.725 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-11T18:46:40.726 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.10 2024-03-11T18:46:41.045 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 systemd[1]: Stopping Ceph osd.10 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:41.045 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:41.045 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:41.045 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:41.045 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:41.046 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:41.046 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:41.046 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:40 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:40.874+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:41.046 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.042+0000 7f1767728700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:41.047 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.042+0000 7f1767728700 -1 osd.10 112 *** Got signal Terminated *** 2024-03-11T18:46:41.047 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.042+0000 7f1767728700 -1 osd.10 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:41.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:41.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:41.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:41.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:41.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:41.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:40 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:40.674+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:42.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:42.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:42.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:42.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:42.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:42.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:41.680+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:42.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:42.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:42.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:42.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:42.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:42.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:42.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:41 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:41.919+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:43.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:43.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:43.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:43.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:43.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:43.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:43.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:42 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:42.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:43.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:43.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:43.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:43.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:43.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:43.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:43.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:42 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:42.915+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:44.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:44.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:44.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:44.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:44.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:44.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:44.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:43 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:43.671+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:44.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:44.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:44.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:44.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:44.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:44.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:44.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:43 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:43.930+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:45.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:45.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:45.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:45.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:45.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:45.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:45.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:44 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:44.689+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:45.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:45.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:45.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:45.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:45.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:45.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:45.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:44 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:44.912+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:46.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:46.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:46.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:46.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:46.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:46.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:46.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:45.686+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:46.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:17.286082+0000 front 2024-03-11T18:45:17.286174+0000 (oldest deadline 2024-03-11T18:45:41.986118+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:27.688511+0000 front 2024-03-11T18:45:27.688476+0000 (oldest deadline 2024-03-11T18:45:52.889313+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:35.190492+0000 front 2024-03-11T18:45:35.190415+0000 (oldest deadline 2024-03-11T18:45:59.290599+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6814 osd.3 since back 2024-03-11T18:45:44.393254+0000 front 2024-03-11T18:45:44.393386+0000 (oldest deadline 2024-03-11T18:46:09.693073+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:54.994153+0000 front 2024-03-11T18:45:54.994179+0000 (oldest deadline 2024-03-11T18:46:16.694376+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:46:02.595568+0000 front 2024-03-11T18:46:02.595440+0000 (oldest deadline 2024-03-11T18:46:27.795921+0000) 2024-03-11T18:46:46.407 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:45 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10[62055]: 2024-03-11T18:46:45.956+0000 7f1763d1c700 -1 osd.10 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:07.796830+0000 front 2024-03-11T18:46:07.796768+0000 (oldest deadline 2024-03-11T18:46:32.496417+0000) 2024-03-11T18:46:46.803 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:46 smithi023 podman[74917]: 2024-03-11 18:46:46.534690715 +0000 UTC m=+5.608913396 container died eb52e5386be340696612c0d17aa77cd1f53fd53563f27397a8a971f63bc51e23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-11T18:46:47.153 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:46 smithi023 podman[74917]: 2024-03-11 18:46:46.900474475 +0000 UTC m=+5.974697134 container cleanup eb52e5386be340696612c0d17aa77cd1f53fd53563f27397a8a971f63bc51e23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T18:46:47.153 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:46 smithi023 bash[74917]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10 2024-03-11T18:46:47.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:47.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:47.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:47.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:47.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:47.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:47.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:46 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:46.673+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:47.406 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[74934]: 2024-03-11 18:46:47.151346532 +0000 UTC m=+0.619614405 container remove eb52e5386be340696612c0d17aa77cd1f53fd53563f27397a8a971f63bc51e23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-11T18:46:47.736 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[75028]: 2024-03-11 18:46:47.384150199 +0000 UTC m=+0.018197288 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:47.737 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[75028]: 2024-03-11 18:46:47.502320077 +0000 UTC m=+0.136367171 container create d1b879677749a1e940300f1969a712e56f040de0865b0827c50d880b13987a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T18:46:47.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:47.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:47.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:47.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:47.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:47.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:47.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:47 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:47.645+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:48.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[75028]: 2024-03-11 18:46:47.864570238 +0000 UTC m=+0.498617320 container init d1b879677749a1e940300f1969a712e56f040de0865b0827c50d880b13987a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T18:46:48.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[75028]: 2024-03-11 18:46:47.867888684 +0000 UTC m=+0.501935759 container start d1b879677749a1e940300f1969a712e56f040de0865b0827c50d880b13987a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10-deactivate, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-11T18:46:48.156 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:47 smithi023 podman[75028]: 2024-03-11 18:46:47.937640211 +0000 UTC m=+0.571687298 container attach d1b879677749a1e940300f1969a712e56f040de0865b0827c50d880b13987a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T18:46:48.467 INFO:journalctl@ceph.osd.10.smithi023.stdout:Mar 11 18:46:48 smithi023 podman[75028]: 2024-03-11 18:46:48.168400746 +0000 UTC m=+0.802447832 container died d1b879677749a1e940300f1969a712e56f040de0865b0827c50d880b13987a14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T18:46:48.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.10.service' 2024-03-11T18:46:48.794 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:48.794 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-11T18:46:48.795 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-11T18:46:48.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.11 2024-03-11T18:46:49.087 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:49.088 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:48.666+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:49.089 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:48 smithi165 systemd[1]: Stopping Ceph osd.11 for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:49.089 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.084+0000 7f6ae21ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:49.089 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.084+0000 7f6ae21ed700 -1 osd.11 112 *** Got signal Terminated *** 2024-03-11T18:46:49.089 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.084+0000 7f6ae21ed700 -1 osd.11 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T18:46:49.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:49.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:49.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:49.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:49.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:49.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:49.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:49 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:49.632+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:51.084 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:51.085 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:50 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:50.659+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:51.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:51.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:51.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:51.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:51.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:51.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:51.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:51 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:51.636+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:53.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:53.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:53.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:53.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:53.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:53.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:53.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:53.157 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:52 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:52.656+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6820 osd.8 since back 2024-03-11T18:46:28.068095+0000 front 2024-03-11T18:46:28.068113+0000 (oldest deadline 2024-03-11T18:46:52.168356+0000) 2024-03-11T18:46:53.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-03-11T18:45:18.458932+0000 front 2024-03-11T18:45:18.458869+0000 (oldest deadline 2024-03-11T18:45:43.758773+0000) 2024-03-11T18:46:53.906 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6804 osd.1 since back 2024-03-11T18:45:25.460157+0000 front 2024-03-11T18:45:25.460104+0000 (oldest deadline 2024-03-11T18:45:51.260829+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6804 osd.2 since back 2024-03-11T18:45:36.462296+0000 front 2024-03-11T18:45:36.462325+0000 (oldest deadline 2024-03-11T18:46:02.362425+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6812 osd.4 since back 2024-03-11T18:45:51.764140+0000 front 2024-03-11T18:45:51.764085+0000 (oldest deadline 2024-03-11T18:46:17.664124+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6812 osd.5 since back 2024-03-11T18:45:58.165028+0000 front 2024-03-11T18:45:58.165090+0000 (oldest deadline 2024-03-11T18:46:24.065151+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.2:6822 osd.6 since back 2024-03-11T18:46:09.966157+0000 front 2024-03-11T18:46:09.966218+0000 (oldest deadline 2024-03-11T18:46:33.465916+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.23:6820 osd.7 since back 2024-03-11T18:46:20.467229+0000 front 2024-03-11T18:46:20.467303+0000 (oldest deadline 2024-03-11T18:46:42.167149+0000) 2024-03-11T18:46:53.907 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:53 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11[62220]: 2024-03-11T18:46:53.631+0000 7f6ade7e1700 -1 osd.11 112 heartbeat_check: no reply from 172.21.15.165:6820 osd.8 since back 2024-03-11T18:46:28.068095+0000 front 2024-03-11T18:46:28.068113+0000 (oldest deadline 2024-03-11T18:46:52.168356+0000) 2024-03-11T18:46:54.847 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:54 smithi165 podman[75095]: 2024-03-11 18:46:54.589692744 +0000 UTC m=+5.600861340 container died d92d46cb35f14566026826bf71214391185fd3ea0bf6f12e10f7b24734d74c6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T18:46:55.130 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:54 smithi165 podman[75095]: 2024-03-11 18:46:54.926996803 +0000 UTC m=+5.938165426 container cleanup d92d46cb35f14566026826bf71214391185fd3ea0bf6f12e10f7b24734d74c6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:46:55.287 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:54 smithi165 bash[75095]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11 2024-03-11T18:46:55.406 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75126]: 2024-03-11 18:46:55.128038915 +0000 UTC m=+0.539451321 container remove d92d46cb35f14566026826bf71214391185fd3ea0bf6f12e10f7b24734d74c6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T18:46:55.791 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75213]: 2024-03-11 18:46:55.520511787 +0000 UTC m=+0.104623268 container create 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-11T18:46:55.792 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75213]: 2024-03-11 18:46:55.433757379 +0000 UTC m=+0.017868876 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T18:46:55.792 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75213]: 2024-03-11 18:46:55.722845267 +0000 UTC m=+0.306956757 container init 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T18:46:55.792 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75213]: 2024-03-11 18:46:55.725988965 +0000 UTC m=+0.310100453 container start 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T18:46:55.792 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:55 smithi165 podman[75213]: 2024-03-11 18:46:55.789445846 +0000 UTC m=+0.373557352 container attach 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T18:46:56.156 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:56 smithi165 podman[75213]: 2024-03-11 18:46:56.017858507 +0000 UTC m=+0.601970001 container died 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T18:46:56.503 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:56 smithi165 podman[75239]: 2024-03-11 18:46:56.484824405 +0000 UTC m=+0.460882408 container remove 302920932cd9afd355bb45c392600199bde844505d187ac3c0ade9dae7343f4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T18:46:56.503 INFO:journalctl@ceph.osd.11.smithi165.stdout:Mar 11 18:46:56 smithi165 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-11T18:46:56.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@osd.11.service' 2024-03-11T18:46:56.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:46:56.546 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-11T18:46:56.546 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-11T18:46:56.547 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.a 2024-03-11T18:46:57.003 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:56 smithi002 systemd[1]: Stopping Ceph mds.a for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:46:57.504 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:57 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a[73930]: 2024-03-11T18:46:57.172+0000 7f5f72000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:46:57.504 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:57 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a[73930]: 2024-03-11T18:46:57.172+0000 7f5f72000700 -1 mds.a *** got signal Terminated *** 2024-03-11T18:46:59.223 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:58 smithi002 podman[95701]: 2024-03-11 18:46:58.905952902 +0000 UTC m=+1.855216615 container died 7139c4795fc322cad0a35d1e36c0b8471367b1dd1e20c5e4035ed3e08db846f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:46:59.503 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:59 smithi002 podman[95701]: 2024-03-11 18:46:59.349147981 +0000 UTC m=+2.298411717 container cleanup 7139c4795fc322cad0a35d1e36c0b8471367b1dd1e20c5e4035ed3e08db846f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T18:46:59.503 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:59 smithi002 bash[95701]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a 2024-03-11T18:47:00.003 INFO:journalctl@ceph.mds.a.smithi002.stdout:Mar 11 18:46:59 smithi002 podman[95715]: 2024-03-11 18:46:59.608741351 +0000 UTC m=+0.690600727 container remove 7139c4795fc322cad0a35d1e36c0b8471367b1dd1e20c5e4035ed3e08db846f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T18:47:00.187 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.a.service' 2024-03-11T18:47:00.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:00.313 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-11T18:47:00.313 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-11T18:47:00.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.d 2024-03-11T18:47:00.753 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:00 smithi002 systemd[1]: Stopping Ceph mds.d for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:01.253 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d[72934]: 2024-03-11T18:47:00.952+0000 7fd41d800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:01.253 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:00 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d[72934]: 2024-03-11T18:47:00.952+0000 7fd41d800700 -1 mds.d *** got signal Terminated *** 2024-03-11T18:47:04.394 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:04 smithi002 podman[95820]: 2024-03-11 18:47:04.108131239 +0000 UTC m=+3.293127615 container died 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:47:04.753 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:04 smithi002 podman[95820]: 2024-03-11 18:47:04.519480947 +0000 UTC m=+3.704477364 container cleanup 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:47:04.753 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:04 smithi002 bash[95820]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d 2024-03-11T18:47:05.253 INFO:journalctl@ceph.mds.d.smithi002.stdout:Mar 11 18:47:04 smithi002 podman[95834]: 2024-03-11 18:47:04.829116923 +0000 UTC m=+0.703854278 container remove 9e62b891b5f686db435975b4e774eef7d511f69d8604fd590dda9f4e411016bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=) 2024-03-11T18:47:05.401 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.d.service' 2024-03-11T18:47:05.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:05.516 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-11T18:47:05.516 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-11T18:47:05.516 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.g 2024-03-11T18:47:05.753 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:05 smithi002 systemd[1]: Stopping Ceph mds.g for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:06.503 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g[73542]: 2024-03-11T18:47:06.062+0000 7f58d1200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:06.503 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:06 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g[73542]: 2024-03-11T18:47:06.062+0000 7f58d1200700 -1 mds.g *** got signal Terminated *** 2024-03-11T18:47:10.338 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:10 smithi002 podman[95941]: 2024-03-11 18:47:10.082644067 +0000 UTC m=+4.143531023 container died c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T18:47:10.659 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:10 smithi002 podman[95941]: 2024-03-11 18:47:10.447816997 +0000 UTC m=+4.508703980 container cleanup c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T18:47:10.660 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:10 smithi002 bash[95941]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g 2024-03-11T18:47:11.003 INFO:journalctl@ceph.mds.g.smithi002.stdout:Mar 11 18:47:10 smithi002 podman[95954]: 2024-03-11 18:47:10.65710378 +0000 UTC m=+0.555669338 container remove c1a9f97df2cfd8bf205a963b053db05dcc7029fb44e076858265a40e28e5e988 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-g, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T18:47:11.130 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.g.service' 2024-03-11T18:47:11.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:11.229 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-11T18:47:11.230 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-11T18:47:11.230 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.j 2024-03-11T18:47:11.503 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:11 smithi002 systemd[1]: Stopping Ceph mds.j for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:12.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j[72556]: 2024-03-11T18:47:11.832+0000 7efd3fa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:12.253 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:11 smithi002 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j[72556]: 2024-03-11T18:47:11.832+0000 7efd3fa00700 -1 mds.j *** got signal Terminated *** 2024-03-11T18:47:16.150 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:15 smithi002 podman[96069]: 2024-03-11 18:47:15.853578487 +0000 UTC m=+4.179742263 container died 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T18:47:16.503 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:16 smithi002 podman[96069]: 2024-03-11 18:47:16.267869124 +0000 UTC m=+4.594032896 container cleanup 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T18:47:16.503 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:16 smithi002 bash[96069]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j 2024-03-11T18:47:16.968 INFO:journalctl@ceph.mds.j.smithi002.stdout:Mar 11 18:47:16 smithi002 podman[96084]: 2024-03-11 18:47:16.526973548 +0000 UTC m=+0.660442114 container remove 2d4a34e73cd761e31033f54bcded51d726475b70dc89a5579fbe1555b59648fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-j, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-11T18:47:17.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.j.service' 2024-03-11T18:47:17.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:17.129 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-11T18:47:17.129 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-11T18:47:17.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.b 2024-03-11T18:47:17.435 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:17 smithi023 systemd[1]: Stopping Ceph mds.b for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:17.715 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: 2024-03-11T18:47:17.433+0000 7ff710841700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:17.715 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:17 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b[67759]: 2024-03-11T18:47:17.433+0000 7ff710841700 -1 mds.b *** got signal Terminated *** 2024-03-11T18:47:20.656 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:20 smithi023 podman[75134]: 2024-03-11 18:47:20.387195754 +0000 UTC m=+3.062593993 container died 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T18:47:21.127 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:20 smithi023 podman[75134]: 2024-03-11 18:47:20.882604832 +0000 UTC m=+3.558003062 container cleanup 2bd0270a9338b0a9dc1f5941b627e66feaf8eeedbf67e9e01a33610e0e368364 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:47:21.127 INFO:journalctl@ceph.mds.b.smithi023.stdout:Mar 11 18:47:20 smithi023 bash[75134]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-b 2024-03-11T18:47:21.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.b.service' 2024-03-11T18:47:21.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:21.313 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-11T18:47:21.314 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-11T18:47:21.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.e 2024-03-11T18:47:21.647 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:21 smithi023 systemd[1]: Stopping Ceph mds.e for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:21.906 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: 2024-03-11T18:47:21.644+0000 7f9450891700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:21.906 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:21 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e[67140]: 2024-03-11T18:47:21.644+0000 7f9450891700 -1 mds.e *** got signal Terminated *** 2024-03-11T18:47:24.656 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:24 smithi023 podman[75268]: 2024-03-11 18:47:24.383889366 +0000 UTC m=+2.840730074 container died d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:47:25.156 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:24 smithi023 podman[75268]: 2024-03-11 18:47:24.752919709 +0000 UTC m=+3.209760411 container cleanup d5c02e53f71c5788b348f387e490bfbeb75532c900afea3ab7ee8e53f75221ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T18:47:25.156 INFO:journalctl@ceph.mds.e.smithi023.stdout:Mar 11 18:47:24 smithi023 bash[75268]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-e 2024-03-11T18:47:25.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.e.service' 2024-03-11T18:47:25.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:25.368 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-11T18:47:25.368 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-11T18:47:25.368 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.h 2024-03-11T18:47:25.792 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:25 smithi023 systemd[1]: Stopping Ceph mds.h for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:26.156 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: 2024-03-11T18:47:25.789+0000 7f00e1d2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:26.156 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:25 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h[67452]: 2024-03-11T18:47:25.789+0000 7f00e1d2e700 -1 mds.h *** got signal Terminated *** 2024-03-11T18:47:28.656 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:28 smithi023 podman[75393]: 2024-03-11 18:47:28.387797983 +0000 UTC m=+2.799598582 container died 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T18:47:29.018 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:28 smithi023 podman[75393]: 2024-03-11 18:47:28.781676968 +0000 UTC m=+3.193477529 container cleanup 829a15f74664f83fe6a5278c3832d5b5dda3d9e6c0d926f5b31e78b934e04f65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T18:47:29.018 INFO:journalctl@ceph.mds.h.smithi023.stdout:Mar 11 18:47:28 smithi023 bash[75393]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-h 2024-03-11T18:47:29.159 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.h.service' 2024-03-11T18:47:29.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:29.197 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-11T18:47:29.197 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-11T18:47:29.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.k 2024-03-11T18:47:29.529 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:29 smithi023 systemd[1]: Stopping Ceph mds.k for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:29.906 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k[66819]: 2024-03-11T18:47:29.527+0000 7f948ecde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:29.906 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:29 smithi023 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k[66819]: 2024-03-11T18:47:29.527+0000 7f948ecde700 -1 mds.k *** got signal Terminated *** 2024-03-11T18:47:32.330 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 podman[75516]: 2024-03-11 18:47:32.0713121 +0000 UTC m=+2.655290551 container died dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-94a5397, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T18:47:32.597 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 podman[75516]: 2024-03-11 18:47:32.419940746 +0000 UTC m=+3.003919177 container cleanup dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T18:47:32.597 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 bash[75516]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k 2024-03-11T18:47:32.847 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 podman[75530]: 2024-03-11 18:47:32.594717447 +0000 UTC m=+0.521084036 container remove dbda0c850c9e3b1de18b10e55a201262123b139bcb42e0f2db5b0760f02ed0da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T18:47:32.847 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-11T18:47:32.848 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 systemd[1]: Stopped Ceph mds.k for 02b83714-dfd4-11ee-95c8-87774f69a715. 2024-03-11T18:47:32.848 INFO:journalctl@ceph.mds.k.smithi023.stdout:Mar 11 18:47:32 smithi023 systemd[1]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.k.service: Consumed 1.081s CPU time. 2024-03-11T18:47:32.849 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.k.service' 2024-03-11T18:47:32.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:32.886 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-11T18:47:32.887 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-11T18:47:32.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.c 2024-03-11T18:47:33.261 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:32 smithi165 systemd[1]: Stopping Ceph mds.c for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:33.656 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: 2024-03-11T18:47:33.258+0000 7f0779db7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:33.656 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:33 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c[67369]: 2024-03-11T18:47:33.258+0000 7f0779db7700 -1 mds.c *** got signal Terminated *** 2024-03-11T18:47:36.761 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:36 smithi165 podman[75319]: 2024-03-11 18:47:36.405585109 +0000 UTC m=+3.324672328 container died b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True) 2024-03-11T18:47:37.027 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:36 smithi165 podman[75319]: 2024-03-11 18:47:36.816251294 +0000 UTC m=+3.735338503 container cleanup b3c934914db8250c096d4d465db1a97e3131645cefe90294959508b38cd4064f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-11T18:47:37.028 INFO:journalctl@ceph.mds.c.smithi165.stdout:Mar 11 18:47:36 smithi165 bash[75319]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-c 2024-03-11T18:47:37.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.c.service' 2024-03-11T18:47:37.274 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:37.274 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-11T18:47:37.274 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-11T18:47:37.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.f 2024-03-11T18:47:37.656 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:37 smithi165 systemd[1]: Stopping Ceph mds.f for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:37.656 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: 2024-03-11T18:47:37.569+0000 7f226feca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:37.656 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:37 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f[67059]: 2024-03-11T18:47:37.569+0000 7f226feca700 -1 mds.f *** got signal Terminated *** 2024-03-11T18:47:40.698 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:40 smithi165 podman[75452]: 2024-03-11 18:47:40.408384195 +0000 UTC m=+2.920639700 container died b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-11T18:47:41.007 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:40 smithi165 podman[75452]: 2024-03-11 18:47:40.786876499 +0000 UTC m=+3.299131987 container cleanup b41286ffee332cb2121adcfb63d3a6857c014eeecfc16659492f08a9ad47fa80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-11T18:47:41.007 INFO:journalctl@ceph.mds.f.smithi165.stdout:Mar 11 18:47:40 smithi165 bash[75452]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-f 2024-03-11T18:47:41.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.f.service' 2024-03-11T18:47:41.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:41.172 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-11T18:47:41.173 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-11T18:47:41.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.i 2024-03-11T18:47:41.406 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:41 smithi165 systemd[1]: Stopping Ceph mds.i for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:41.906 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: 2024-03-11T18:47:41.508+0000 7f2dd5efd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:41.906 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:41 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i[67686]: 2024-03-11T18:47:41.508+0000 7f2dd5efd700 -1 mds.i *** got signal Terminated *** 2024-03-11T18:47:44.602 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:44 smithi165 podman[75572]: 2024-03-11 18:47:44.384295553 +0000 UTC m=+2.999616780 container died e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T18:47:44.906 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:44 smithi165 podman[75572]: 2024-03-11 18:47:44.682285117 +0000 UTC m=+3.297606328 container cleanup e2e6bccfd5d609cf75b6c30ef45dfdb522d7f81e8bc48e99753597f168737703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T18:47:44.906 INFO:journalctl@ceph.mds.i.smithi165.stdout:Mar 11 18:47:44 smithi165 bash[75572]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-i 2024-03-11T18:47:45.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.i.service' 2024-03-11T18:47:45.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:45.133 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-11T18:47:45.133 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-11T18:47:45.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.l 2024-03-11T18:47:45.463 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:45 smithi165 systemd[1]: Stopping Ceph mds.l for 02b83714-dfd4-11ee-95c8-87774f69a715... 2024-03-11T18:47:45.464 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l[66749]: 2024-03-11T18:47:45.460+0000 7f208c34a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T18:47:45.464 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:45 smithi165 ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l[66749]: 2024-03-11T18:47:45.460+0000 7f208c34a700 -1 mds.l *** got signal Terminated *** 2024-03-11T18:47:49.096 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:48 smithi165 podman[75692]: 2024-03-11 18:47:48.788320533 +0000 UTC m=+3.441263847 container died 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T18:47:49.096 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:49 smithi165 podman[75692]: 2024-03-11 18:47:49.094312522 +0000 UTC m=+3.747255819 container cleanup 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T18:47:49.406 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:49 smithi165 bash[75692]: ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l 2024-03-11T18:47:49.406 INFO:journalctl@ceph.mds.l.smithi165.stdout:Mar 11 18:47:49 smithi165 podman[75706]: 2024-03-11 18:47:49.311930804 +0000 UTC m=+0.518077140 container remove 2ac009db11b6883bec7897dc6e013fbd956ee1deb0b3645a24e8f3e48ad0d84b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-02b83714-dfd4-11ee-95c8-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T18:47:49.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-02b83714-dfd4-11ee-95c8-87774f69a715@mds.l.service' 2024-03-11T18:47:49.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T18:47:49.476 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-11T18:47:49.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T18:47:49.814 INFO:teuthology.orchestra.run.smithi002.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:47:54.756 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T18:47:54.958 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:47:57.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T18:47:57.832 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:48:00.285 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:48:00.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:48:00.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T18:48:00.465 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T18:48:00.467 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi002/crash 2024-03-11T18:48:00.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T18:48:00.596 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T18:48:00.596 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2024-03-11T18:48:00.603 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi023/crash 2024-03-11T18:48:00.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T18:48:00.783 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T18:48:00.783 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-11T18:48:00.786 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi165/crash 2024-03-11T18:48:00.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash -- . 2024-03-11T18:48:00.822 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T18:48:00.823 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-03-11T18:48:00.824 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T18:48:00.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T18:48:00.932 INFO:teuthology.orchestra.run.smithi002.stdout:2024-03-11T18:23:36.179006+0000 mon.a (mon.0) 189 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T18:48:00.933 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T18:48:00.933 INFO:tasks.cephadm:Compressing logs... 2024-03-11T18:48:00.933 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T18:48:00.938 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T18:48:00.940 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T18:48:00.981 INFO:teuthology.orchestra.run.smithi165.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T18:48:00.982 INFO:teuthology.orchestra.run.smithi165.stderr:: No such file or directory 2024-03-11T18:48:00.982 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.l.log 2024-03-11T18:48:00.982 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T18:48:00.982 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.l.log: --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.i.log 2024-03-11T18:48:00.983 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.11.log 2024-03-11T18:48:00.983 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T18:48:00.983 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.i.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.8.log 2024-03-11T18:48:00.984 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.z.log: 92.4%gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T18:48:00.984 INFO:teuthology.orchestra.run.smithi165.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T18:48:00.984 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.f.log 2024-03-11T18:48:00.984 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T18:48:00.985 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.5.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.f.log: 90.1% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T18:48:00.985 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T18:48:00.985 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.3% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T18:48:00.986 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T18:48:00.986 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log: 94.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.l.log.gz 2024-03-11T18:48:00.986 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.c.log 2024-03-11T18:48:00.986 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.2.log: 86.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T18:48:00.987 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T18:48:00.987 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.c.log: 92.0% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T18:48:00.987 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.10.log 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.b.log 2024-03-11T18:48:00.993 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.k.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.h.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.e.log 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.7.log: 90.1% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T18:48:00.994 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T18:48:00.995 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.e.log: 94.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.k.log.gz 2024-03-11T18:48:00.995 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T18:48:00.995 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.y.log: 91.9% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T18:48:00.996 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T18:48:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T18:48:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log: 83.3% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T18:48:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T18:48:00.999 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log: 86.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T18:48:00.999 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T18:48:01.009 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T18:48:01.010 INFO:teuthology.orchestra.run.smithi002.stderr:: No such file or directory 2024-03-11T18:48:01.010 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T18:48:01.012 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T18:48:01.013 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T18:48:01.013 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T18:48:01.014 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.g.log 2024-03-11T18:48:01.015 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.j.log 2024-03-11T18:48:01.017 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T18:48:01.019 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T18:48:01.021 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.3.log: 90.0% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T18:48:01.021 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T18:48:01.022 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.x.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T18:48:01.023 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.0.log: 94.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.g.log.gz 2024-03-11T18:48:01.025 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T18:48:01.025 INFO:teuthology.orchestra.run.smithi002.stderr: 94.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.j.log.gz 2024-03-11T18:48:01.026 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log 2024-03-11T18:48:01.028 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T18:48:01.028 INFO:teuthology.orchestra.run.smithi002.stderr: 83.8% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T18:48:01.031 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log: /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.9.log 2024-03-11T18:48:01.034 INFO:teuthology.orchestra.run.smithi002.stderr: 86.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T18:48:01.035 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.d.log 2024-03-11T18:48:01.038 INFO:teuthology.orchestra.run.smithi165.stderr: 94.6% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.f.log.gz 2024-03-11T18:48:01.043 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.9.log: gzip -5 --verbose/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.d.log: -- /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.a.log 2024-03-11T18:48:01.045 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-11T18:48:01.054 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 94.3% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.d.log.gz 2024-03-11T18:48:01.062 INFO:teuthology.orchestra.run.smithi002.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-11T18:48:01.063 INFO:teuthology.orchestra.run.smithi002.stderr: 90.9% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T18:48:01.064 INFO:teuthology.orchestra.run.smithi002.stderr: 94.3% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.a.log.gz 2024-03-11T18:48:01.308 INFO:teuthology.orchestra.run.smithi002.stderr: 89.6% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T18:48:01.579 INFO:teuthology.orchestra.run.smithi165.stderr: 91.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T18:48:01.981 INFO:teuthology.orchestra.run.smithi023.stderr: 92.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T18:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr: 95.1% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.e.log.gz 2024-03-11T18:48:02.391 INFO:teuthology.orchestra.run.smithi002.stderr: 90.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T18:48:02.542 INFO:teuthology.orchestra.run.smithi165.stderr: 96.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.i.log.gz 2024-03-11T18:48:06.870 INFO:teuthology.orchestra.run.smithi165.stderr: 95.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.c.log.gz 2024-03-11T18:48:17.675 INFO:teuthology.orchestra.run.smithi165.stderr: 91.6% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.11.log.gz 2024-03-11T18:48:18.270 INFO:teuthology.orchestra.run.smithi002.stderr: 91.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T18:48:18.585 INFO:teuthology.orchestra.run.smithi165.stderr: 91.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.8.log.gz 2024-03-11T18:48:20.460 INFO:teuthology.orchestra.run.smithi165.stderr: 91.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T18:48:20.672 INFO:teuthology.orchestra.run.smithi002.stderr: 91.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T18:48:21.616 INFO:teuthology.orchestra.run.smithi023.stderr: 91.7% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.10.log.gz 2024-03-11T18:48:21.818 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T18:48:22.140 INFO:teuthology.orchestra.run.smithi023.stderr: 94.1% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.h.log.gz 2024-03-11T18:48:22.241 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T18:48:24.067 INFO:teuthology.orchestra.run.smithi023.stderr: 91.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T18:48:24.193 INFO:teuthology.orchestra.run.smithi023.stderr: 91.9% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T18:48:24.549 INFO:teuthology.orchestra.run.smithi002.stderr: 91.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.9.log.gz 2024-03-11T18:48:24.554 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-03-11T18:48:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m23.581s 2024-03-11T18:48:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:user 1m17.410s 2024-03-11T18:48:24.554 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m5.649s 2024-03-11T18:48:27.733 INFO:teuthology.orchestra.run.smithi165.stderr: 91.4% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T18:48:27.735 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-03-11T18:48:27.735 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m26.780s 2024-03-11T18:48:27.735 INFO:teuthology.orchestra.run.smithi165.stderr:user 1m25.923s 2024-03-11T18:48:27.735 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m2.668s 2024-03-11T18:50:04.449 INFO:teuthology.orchestra.run.smithi023.stderr: 93.5% -- replaced with /var/log/ceph/02b83714-dfd4-11ee-95c8-87774f69a715/ceph-mds.b.log.gz 2024-03-11T18:50:04.451 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-11T18:50:04.451 INFO:teuthology.orchestra.run.smithi023.stderr:real 2m3.495s 2024-03-11T18:50:04.451 INFO:teuthology.orchestra.run.smithi023.stderr:user 3m47.417s 2024-03-11T18:50:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m6.464s 2024-03-11T18:50:04.452 INFO:tasks.cephadm:Archiving logs... 2024-03-11T18:50:04.453 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi002/log 2024-03-11T18:50:04.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T18:50:17.035 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi023/log 2024-03-11T18:50:17.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T18:50:46.882 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi165/log 2024-03-11T18:50:46.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T18:51:00.361 INFO:tasks.cephadm:Removing cluster... 2024-03-11T18:51:00.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force 2024-03-11T18:51:00.709 INFO:teuthology.orchestra.run.smithi002.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:51:01.625 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force 2024-03-11T18:51:01.830 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:51:02.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 02b83714-dfd4-11ee-95c8-87774f69a715 --force 2024-03-11T18:51:02.552 INFO:teuthology.orchestra.run.smithi165.stdout:Deleting cluster with fsid: 02b83714-dfd4-11ee-95c8-87774f69a715 2024-03-11T18:51:02.931 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T18:51:02.932 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T18:51:02.992 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T18:51:03.010 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T18:51:03.025 INFO:tasks.cephadm:Teardown complete 2024-03-11T18:51:03.025 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T18:51:03.037 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T18:51:03.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T18:51:03.045 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T18:51:03.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T18:51:03.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T18:51:03.222 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T18:51:03.222 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:51:03.230 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:51:03.233 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T18:51:03.246 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:51:03.247 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:51:03.251 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:51:03.251 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-03-11T18:51:03.251 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 -453us[ -429us] +/- 124ms 2024-03-11T18:51:03.251 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 -1434us[-1410us] +/- 55ms 2024-03-11T18:51:03.252 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +2242us[+2266us] +/- 74ms 2024-03-11T18:51:03.252 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:51:03.252 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:51:03.253 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-11T18:51:03.253 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -449us[ -449us] +/- 124ms 2024-03-11T18:51:03.253 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -1416us[-1387us] +/- 55ms 2024-03-11T18:51:03.253 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +2270us[+2299us] +/- 74ms 2024-03-11T18:51:03.253 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:51:03.280 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-03-11T18:51:03.291 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T18:51:03.292 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-03-11T18:51:03.292 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -432us[ -432us] +/- 124ms 2024-03-11T18:51:03.292 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -1389us[-1419us] +/- 55ms 2024-03-11T18:51:03.292 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +2304us[+2304us] +/- 74ms 2024-03-11T18:51:03.292 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T18:51:03.293 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T18:51:03.303 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T18:51:03.303 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T18:51:03.314 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T18:51:03.562 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T18:51:03.650 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T18:51:03.723 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:51:03.828 INFO:teuthology.orchestra.run.smithi002.stdout:type=AVC msg=audit(1710180720.227:197): avc: denied { checkpoint_restore } for pid=1163 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-11T18:51:03.830 DEBUG:teuthology.task.selinux:ubuntu@smithi002.front.sepia.ceph.com has 1 denials 2024-03-11T18:51:03.830 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:51:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:51:03.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T18:51:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:51:03.898 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 166, in teardown self.get_new_denials() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 215, in get_new_denials raise SELinuxError(node=remote, teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi002.front.sepia.ceph.com: ['type=AVC msg=audit(1710180720.227:197): avc: denied { checkpoint_restore } for pid=1163 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-11T18:51:03.906 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T18:51:03.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T18:51:03.926 INFO:teuthology.task.internal:Duration was 2344.424265 seconds 2024-03-11T18:51:03.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T18:51:03.936 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T18:51:03.936 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T18:51:03.952 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T18:51:03.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T18:51:03.998 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:51:04.007 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:51:04.112 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T18:51:04.582 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T18:51:04.582 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-03-11T18:51:04.583 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T18:51:04.647 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-11T18:51:04.648 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T18:51:04.678 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-03-11T18:51:04.678 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T18:51:04.705 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T18:51:04.705 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T18:51:04.712 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T18:51:04.721 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T18:51:04.811 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T18:51:04.811 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T18:51:04.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T18:51:04.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T18:51:05.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T18:51:05.341 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T18:51:05.341 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T18:51:05.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T18:51:05.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T18:51:05.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T18:51:05.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.546 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-11T18:51:05.562 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-03-11T18:51:05.627 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-03-11T18:51:05.683 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:51:05.734 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:51:05.749 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T18:51:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T18:51:05.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T18:51:05.776 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T18:51:05.777 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi002 2024-03-11T18:51:05.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T18:51:05.897 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi023 2024-03-11T18:51:05.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T18:51:05.941 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226/remote/smithi165 2024-03-11T18:51:05.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T18:51:05.979 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T18:51:05.979 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T18:51:05.987 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T18:51:05.989 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T18:51:06.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T18:51:06.052 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T18:51:06.053 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T18:51:06.070 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T18:51:06.070 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T18:51:06.078 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T18:51:06.080 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T18:51:06.093 INFO:teuthology.orchestra.run.smithi023.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 18:51 /home/ubuntu/cephtest 2024-03-11T18:51:06.094 INFO:teuthology.orchestra.run.smithi165.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 18:51 /home/ubuntu/cephtest 2024-03-11T18:51:06.131 INFO:teuthology.orchestra.run.smithi002.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 18:51 /home/ubuntu/cephtest 2024-03-11T18:51:06.135 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T18:51:06.170 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T18:51:06.208 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi002.front.sepia.ceph.com: ['type=AVC msg=audit(1710180720.227:197): avc: denied { checkpoint_restore } for pid=1163 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-11T18:51:06.208 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T18:51:06.233 DEBUG:teuthology.nuke:shortname: smithi002 2024-03-11T18:51:06.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T18:51:06.253 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.913386', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEvGf+cyMZ6EdEn2J5RHTQc6jlKSmHOa15J+g16R020D/WnTP8sd+p/577nli/j7ejRB8I2iXIfrbjR9bK02/Ro='} 2024-03-11T18:51:06.259 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-11T18:51:06.259 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T18:51:06.281 DEBUG:teuthology.nuke:shortname: smithi165 2024-03-11T18:51:06.281 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T18:51:06.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.917054', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtVfIZXn/vKDzdE72Fnb01pPZmHprqSHaw9vDG/7pOEIskG0900+I48l30rMFl6AWLRMiO2ef3/L3kCdI0AZ3U='} 2024-03-11T18:51:06.304 INFO:teuthology.orchestra.console.smithi002:Power off 2024-03-11T18:51:06.305 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T18:51:06.321 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_14:02:35-fs-squid-distro-default-smithi/7591226', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 18:02:35.915329', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDICgzGvuK60VEuW89K0LVqhLtrL7SvthPpgehstdTNAi0NW8PNpL1kCZN4Lea83uh8HZQWUTcO9hc//CYBm1sc='} 2024-03-11T18:51:06.332 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2024-03-11T18:51:06.332 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:06.352 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-11T18:51:06.352 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T18:51:06.366 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-03-11T18:51:06.376 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-11T18:51:06.376 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:06.395 INFO:teuthology.orchestra.console.smithi165:Power off 2024-03-11T18:51:06.395 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T18:51:06.405 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-11T18:51:06.419 DEBUG:teuthology.orchestra.console.smithi165:power off output: Chassis Power Control: Down/Off 2024-03-11T18:51:06.419 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:06.442 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-11T18:51:10.367 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:10.406 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:10.443 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:10.479 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-03-11T18:51:10.518 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-11T18:51:10.553 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is on 2024-03-11T18:51:14.482 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:14.520 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:14.554 DEBUG:teuthology.orchestra.console.smithi165:pexpect command: ipmitool -H smithi165.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T18:51:14.593 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2024-03-11T18:51:14.630 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-11T18:51:14.663 DEBUG:teuthology.orchestra.console.smithi165:check power output: Chassis Power is off 2024-03-11T18:51:14.694 INFO:teuthology.orchestra.console.smithi002:Power off completed 2024-03-11T18:51:14.730 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-11T18:51:14.765 INFO:teuthology.orchestra.console.smithi165:Power off completed 2024-03-11T18:51:14.872 INFO:teuthology.run:Summary data: client.0-kernel-sha1: dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2344.4242651462555 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi002 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6a4997be2b8bb74411268cc42753f29ae9a78ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1b4633bc9037418db2cde4842135fe40 status: fail success: false 2024-03-11T18:51:14.872 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T18:51:14.980 INFO:teuthology.run:FAIL